From d159e57261ab621089c781a020b235c7afd7b182 Mon Sep 17 00:00:00 2001 From: Techassi Date: Fri, 7 Nov 2025 12:25:47 +0100 Subject: [PATCH 01/54] chore: Initial commit --- modules/ROOT/pages/release-notes.adoc | 2 ++ modules/ROOT/partials/release-notes/release-25.11.adoc | 6 ++++++ 2 files changed, 8 insertions(+) create mode 100644 modules/ROOT/partials/release-notes/release-25.11.adoc diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index 2212ed303..5f89b55d1 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -7,6 +7,8 @@ The Stackable Data Platform consists of multiple operators that work together. Periodically a platform release is made, including all components of the platform at a specific version. // WARNING: Please keep the empty newlines, otherwise headings are broken. +include::partial$release-notes/release-25.11.adoc[] + include::partial$release-notes/release-25.7.adoc[] include::partial$release-notes/release-25.3.adoc[] diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc new file mode 100644 index 000000000..cb7a444bd --- /dev/null +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -0,0 +1,6 @@ +== Release 25.11 + +=== 25.11.0 + +Released on YYYY-MM-DD. +(Optional description / introduction) From 07cd087889bdbb0c55019294a2438344d2eec597 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:29:34 +0100 Subject: [PATCH 02/54] docs: Add new general platform features --- .../partials/release-notes/release-25.11.adoc | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index cb7a444bd..3019c2cac 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -4,3 +4,26 @@ Released on YYYY-MM-DD. (Optional description / introduction) + +==== New platform features + +===== General + +====== Custom image selection + +Previously, when using custom images in combination with a SHA digest like `oci.stackable.tech/sdp/spark-k8s@sha256:c8b7...`, all operators created invalid labels `app.kubernetes.io/version` for their applied resources. +This was fixed by checking and replacing invalid characters in the created labels when a SHA digest is used to select the custom image. +See https://github.com/stackabletech/operator-rs/pull/1076[operator-rs#1076]. + +====== Security + +Traffic between Open Policy Agent (OPA) and clients can be encrypted using TLS by enabling it in the OPA custom resource. +The authorizers for Trino and NiFi automatically integrate with these secured OPA deployments and verify the authenticity of the server certificates when TLS for OPA is enabled. +See the xref:opa:usage-guide/tls.adoc[TLS encryption documentation page] and https://github.com/stackabletech/opa-operator/issues/581[opa-operator#581]. + +====== Miscellaneous + +// TODO: Do we want to include this? +The performance of the Trino rules in the `end-to-end-security` stack was improved. +Batch queries are now significantly faster. +See https://github.com/stackabletech/demos/pull/289[demos#289]. From 0b67cc78d24f8e47810c824e800235635896da5d Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:30:16 +0100 Subject: [PATCH 03/54] docs: Add new Airflow features --- modules/ROOT/partials/release-notes/release-25.11.adoc | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 3019c2cac..6a7b748c0 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -27,3 +27,11 @@ See the xref:opa:usage-guide/tls.adoc[TLS encryption documentation page] and htt The performance of the Trino rules in the `end-to-end-security` stack was improved. Batch queries are now significantly faster. See https://github.com/stackabletech/demos/pull/289[demos#289]. + +===== Apache Airflow + +* The Airflow xref:airflow:index.adoc#_triggerers[triggerer] component is now supported. + This can be used with DAGs utilising deferrable operators to keep worker slots free and enhance HA. + See https://github.com/stackabletech/airflow-operator/issues/200[airflow-operator#200]. +* The xref:demos:airflow-scheduled-job.adoc[`airflow-scheduled-job`] demo for Airflow has been extended to showcase some of the new Airflow 3.x features in the context of SDP i.e. event scheduling (with Kafka), triggerer actions and user authorization with OPA and the SDP OPA authorizer. + See https://github.com/stackabletech/demos/issues/223[demos#223]. From e64b45433edf5ca51fb56192f0b352ec1299519b Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:30:36 +0100 Subject: [PATCH 04/54] docs: Add new NiFi features --- modules/ROOT/partials/release-notes/release-25.11.adoc | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 6a7b748c0..ba06246aa 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -35,3 +35,10 @@ See https://github.com/stackabletech/demos/pull/289[demos#289]. See https://github.com/stackabletech/airflow-operator/issues/200[airflow-operator#200]. * The xref:demos:airflow-scheduled-job.adoc[`airflow-scheduled-job`] demo for Airflow has been extended to showcase some of the new Airflow 3.x features in the context of SDP i.e. event scheduling (with Kafka), triggerer actions and user authorization with OPA and the SDP OPA authorizer. See https://github.com/stackabletech/demos/issues/223[demos#223]. + +===== Apache NiFi + +A patch was added which allows disabling the SNI (Server Name Indication) checks for NiFi. +The workaround is documented in the xref:nifi:troubleshooting/index.adoc[troubleshooting] section. +This can be useful in certain scenarios where the external name is not in the certificates used by NiFi. +See https://github.com/stackabletech/nifi-operator/issues/812[nifi-operator#812]. From 621b83f188cc2b18542fdc3fc0398bc52edde4dc Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:31:50 +0100 Subject: [PATCH 05/54] docs: Add new Spark features --- modules/ROOT/partials/release-notes/release-25.11.adoc | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index ba06246aa..b6542fddf 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -42,3 +42,13 @@ A patch was added which allows disabling the SNI (Server Name Indication) checks The workaround is documented in the xref:nifi:troubleshooting/index.adoc[troubleshooting] section. This can be useful in certain scenarios where the external name is not in the certificates used by NiFi. See https://github.com/stackabletech/nifi-operator/issues/812[nifi-operator#812]. + +===== Apache Spark + +* The service account of spark applications can now be overridden with pod overrides. + Previously, the application service account was passed as command line argument to spark-submit and it was therefor not possible to overwrite it with pod overrides for the driver and executors. + This CLI argument has now been moved to the pod templates of the individual roles. + See https://github.com/stackabletech/spark-k8s-operator/pull/617[spark-k8s-operator#617]. +* This release adds experimental support for Spark 4.X.X. + The support is marked as experimental because Spark 4.0.0 has known compatibility issues with https://github.com/apache/hbase-connectors/pull/130[Apache HBase] and https://github.com/apache/iceberg/issues/13358[Apache Iceberg]. + See https://github.com/stackabletech/spark-k8s-operator/issues/586[spark-k8s-operator#586]. From b9f0dd744cd2390166cd90e78dcd78231b4d8eb7 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:32:15 +0100 Subject: [PATCH 06/54] docs: Add new OPA features --- modules/ROOT/partials/release-notes/release-25.11.adoc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index b6542fddf..853119271 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -52,3 +52,9 @@ See https://github.com/stackabletech/nifi-operator/issues/812[nifi-operator#812] * This release adds experimental support for Spark 4.X.X. The support is marked as experimental because Spark 4.0.0 has known compatibility issues with https://github.com/apache/hbase-connectors/pull/130[Apache HBase] and https://github.com/apache/iceberg/issues/13358[Apache Iceberg]. See https://github.com/stackabletech/spark-k8s-operator/issues/586[spark-k8s-operator#586]. + +====== Open Policy Agent + +This release adds a dedicated per-rolegroup `-metrics` Service, which can be used to scrape Prometheus metrics. +// TODO: How do we expose more metrics? The PR this snippet comes from doesn't seem to expose more metrics. +// The Expose more Prometheus metrics, such as successful or failed bundle loads and information about the OPA environment From cf464c7810497019e4cd783289708cea3ffa892f Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:32:40 +0100 Subject: [PATCH 07/54] docs: Add new Trino features --- modules/ROOT/partials/release-notes/release-25.11.adoc | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 853119271..16a5994e0 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -58,3 +58,13 @@ See https://github.com/stackabletech/nifi-operator/issues/812[nifi-operator#812] This release adds a dedicated per-rolegroup `-metrics` Service, which can be used to scrape Prometheus metrics. // TODO: How do we expose more metrics? The PR this snippet comes from doesn't seem to expose more metrics. // The Expose more Prometheus metrics, such as successful or failed bundle loads and information about the OPA environment + +====== Trino + +* The operator now supports configuring fault-tolerant execution for Trino via the TrinoCluster CRD. + See the xref:trino:usage-guide/fault-tolerant-execution.adoc[documentation page] and https://github.com/stackabletech/trino-operator/pull/779[trino-operator#779]. +* The Trino client spooling protocol can now be configured using the `spec.clusterConfig.clientProtocol.spooling` property. +// TODO: Is this an S3Connection (CR)? + Users can configure an S3 connection and the location or spooling segments. + Additional properties can be added using the `configOverrides` mechanism for the `spooling-manager.properties` file. + See the xref:trino:usage-guide/client-spooling-protocol.adoc[client spooling protocol] documentation page and https://github.com/stackabletech/trino-operator/pull/793[trino-operator#793]. From 5605fcebe5387108cde548fb4f365707b559179c Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:33:15 +0100 Subject: [PATCH 08/54] docs: Add general platform improvements --- .../partials/release-notes/release-25.11.adoc | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 16a5994e0..26fd5a737 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -68,3 +68,60 @@ This release adds a dedicated per-rolegroup `-metrics` Service, which can be use Users can configure an S3 connection and the location or spooling segments. Additional properties can be added using the `configOverrides` mechanism for the `spooling-manager.properties` file. See the xref:trino:usage-guide/client-spooling-protocol.adoc[client spooling protocol] documentation page and https://github.com/stackabletech/trino-operator/pull/793[trino-operator#793]. + +==== Platform improvements + +===== Vulnerabilities + +37 CVEs were fixed in the Stackable product images. +This includes 2 critical and 18 high-severity CVEs. + +===== General + +====== Observability + +This release includes various improvements in regards to metrics collection and exposition. +Previously, some operators did not expose Prometheus annotations containing the HTTP(S) scheme or the metrics path and port. +These annotations are now available which allows custom relabel configs in Prometheus to scrape the metric endpoints. + +* Apache HBase: The `prometheus.io/scrape` label is now only available on the `metrics` Service (instead of the `headless` service), which uses `metrics` as the port name instead of the previous `ui-http`/`ui-https` port name. + See https://github.com/stackabletech/hbase-operator/pull/701[hbase-operator#701]. +* Apache Airflow: The operator now adds the appropriate Prometheus annotations. + See https://github.com/stackabletech/airflow-operator/pull/698[airflow-operator#698]. +* Apache Druid: The operator now adds the appropriate Prometheus annotations. + See https://github.com/stackabletech/druid-operator/pull/761[airflow-operator#761]. + +====== Miscellaneous + +* All operators now correctly handle multiple CA certificates. + This can be the case if the Stackable secret-operator auto rotated the CA certificate or if multiple CA certificates are present in a SecretClass. + See https://github.com/stackabletech/issues/issues/764[issues#764] for more details. +* New Helm values have been added to the operators for setting `priorityClassName` on the resulting Pods, giving administrators greater control over scheduling. + When left unconfigured, the fields will not be present on the subsequent Pods. + See https://github.com/stackabletech/issues/issues/765[issues#765] for more details. ++ +[source,yaml] +---- +# Listener operator +csiProvisioner: + priorityClassName: ... + +csiNodeDriver: + priorityClassName: ... + +# TODO: List/check secret-operator here as well +# All other operators +priorityClassName: ... +---- + +* Previously, log entries for some supported products were broken from time to time. + These issues have now been resolved by implementing multiple fixes in various affected (upstream) projects. + See the tracking issue https://github.com/stackabletech/issues/issues/778[issues#778] for more details. +** Pull request https://github.com/vectordotdev/vector/pull/24028[vectordotdev/vector#24028] was raised to fix log entries with multi-char delimiters. + At the time of writing, this PR has not been merged yet, but the fix is manually applied as a patch. + See https://github.com/stackabletech/docker-images/pull/1323[docker-images#1323]. +** A XMLLayout multithreading issue in logback has been fixed by raising https://github.com/qos-ch/logback/pull/978[qos-ch/logback#978]. + This fix has been rolled out in all affected products: +*** Apache Kafka: https://github.com/stackabletech/docker-images/pull/1330[docker-images#1330] +*** Apache NiFi: https://github.com/stackabletech/docker-images/pull/1314[docker-images#1314] +*** Apache ZooKeeper: https://github.com/stackabletech/docker-images/pull/1320[docker-images#1320] From 7073fc4357f49f0d1019c61e71659cafb09d7796 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:34:30 +0100 Subject: [PATCH 09/54] docs: Add Airflow improvements --- .../partials/release-notes/release-25.11.adoc | 29 +++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 26fd5a737..29ccb99fa 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -125,3 +125,32 @@ priorityClassName: ... *** Apache Kafka: https://github.com/stackabletech/docker-images/pull/1330[docker-images#1330] *** Apache NiFi: https://github.com/stackabletech/docker-images/pull/1314[docker-images#1314] *** Apache ZooKeeper: https://github.com/stackabletech/docker-images/pull/1320[docker-images#1320] + +===== Apache Airflow + +* Previously, a missing OPA ConfigMap would crash the operator. + With this release, we don't panic on an invalid authorization config. + See https://github.com/stackabletech/airflow-operator/pull/667[airflow-operator#667]. +* Previously, OPA authorization for Airflow 3 was not working. + With this release, the operator now sets the required environment variables. + See https://github.com/stackabletech/airflow-operator/pull/668[airflow-operator#668]. +* Allow multiple Airflows in the same namespace to use Kubernetes executors. + Previously, the operator would always use the same name for the executor Pod template ConfigMap. + Thus when deploying multiple Airflow instances in the same namespace, the ConfigMaps would conflict. + See https://github.com/stackabletech/airflow-operator/pull/678[airflow-operator#678]. +// TODO: Is there only ONE key or should we clarify WHICH key this is? +* The JWT key is now created by the the operator internally. + The same applies to the key previously defined in the credentials secret under `connections.secretKey`: this change is non-breaking, as `connections.secretKey` will be ignored if supplied. + See https://github.com/stackabletech/airflow-operator/pull/686[airflow-operator#686]. +* Database initialization routines - which are idempotent and run by default - can be deactivated to e.g. help diagnose or troubleshoot start-up issues via the new `databaseInitialization.enabled` field. ++ +[WARNING] +==== +Turning off these routines is an unsupported operation as subsequent updates to a running Airflow cluster can result in broken behaviour due to inconsistent metadata. +Only use this setting if you know what you are doing! +==== +* The Airflow xref:airflow:index.adoc#_dag_processors[DAG-processor] component now has an optional individual role in the resource definition, allowing it to be separately configured (e.g. logging, resources) and run in a dedicated container. + See https://github.com/stackabletech/airflow-operator/issues/637[airflow-operator#637]. +* Previously in set-ups where multiple Web-/API-servers were used, only one instance was able to automatically access the connection passwords stored in the database. + This could be solved by setting the https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#fernet-key[fernet] key explicitly, but now this detail is taken care of internally by the operator. + See https://github.com/stackabletech/airflow-operator/issues/694[airflow-operator#694]. From 308e7387c753d69e1168b24e26f562fffea5d1a6 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:34:54 +0100 Subject: [PATCH 10/54] docs: Add NiFI improvements --- modules/ROOT/partials/release-notes/release-25.11.adoc | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 29ccb99fa..ee6fae45d 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -154,3 +154,8 @@ Only use this setting if you know what you are doing! * Previously in set-ups where multiple Web-/API-servers were used, only one instance was able to automatically access the connection passwords stored in the database. This could be solved by setting the https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#fernet-key[fernet] key explicitly, but now this detail is taken care of internally by the operator. See https://github.com/stackabletech/airflow-operator/issues/694[airflow-operator#694]. + +===== Apache NiFi + +The Apache NiFi xref:nifi:usage_guide/monitoring.adoc#_configure_metrics_in_nifi_2_x_x[monitoring documentation] page has been updated to include guidance on how to scrape NiFi 2.X.X metrics using mTLS. +See https://github.com/stackabletech/nifi-operator/issues/813[nifi-operator#813]. From 3655e87756c19ce3d33a33772c35cc1845eb51d5 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:35:17 +0100 Subject: [PATCH 11/54] docs: Add Spark improvements --- modules/ROOT/partials/release-notes/release-25.11.adoc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index ee6fae45d..84b6b534e 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -159,3 +159,9 @@ Only use this setting if you know what you are doing! The Apache NiFi xref:nifi:usage_guide/monitoring.adoc#_configure_metrics_in_nifi_2_x_x[monitoring documentation] page has been updated to include guidance on how to scrape NiFi 2.X.X metrics using mTLS. See https://github.com/stackabletech/nifi-operator/issues/813[nifi-operator#813]. + +===== Apache Spark + +Spark Connect: Previously the property `spec.image.pullSecrets` was ignored by the operator when creating the executor templates. +This has now been corrected in the operator code. +See https://github.com/stackabletech/spark-k8s-operator/issues/600[spark-k8s-operator#600]. From 970a6f90ed5a35b88e7896c6f5607461ef8f1c87 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:35:37 +0100 Subject: [PATCH 12/54] docs: Add Superset improvements --- modules/ROOT/partials/release-notes/release-25.11.adoc | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 84b6b534e..fa82f38d6 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -165,3 +165,9 @@ See https://github.com/stackabletech/nifi-operator/issues/813[nifi-operator#813] Spark Connect: Previously the property `spec.image.pullSecrets` was ignored by the operator when creating the executor templates. This has now been corrected in the operator code. See https://github.com/stackabletech/spark-k8s-operator/issues/600[spark-k8s-operator#600]. + +===== Apache Superset + +Previously, there was a chance containers would not start, because Superset was starting too slowly and was killed because of a failing liveness probe. +Now, we add a proper startup probe, which allows Superset startup to succeed and not be killed. +See https://github.com/stackabletech/superset-operator/pull/654[superset-operator#654]. From f9b9bb5b4ce981339a4134e160e1b4dc257047d2 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:36:09 +0100 Subject: [PATCH 13/54] docs: Add OPA improvements --- modules/ROOT/partials/release-notes/release-25.11.adoc | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index fa82f38d6..c9f70a37f 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -171,3 +171,13 @@ See https://github.com/stackabletech/spark-k8s-operator/issues/600[spark-k8s-ope Previously, there was a chance containers would not start, because Superset was starting too slowly and was killed because of a failing liveness probe. Now, we add a proper startup probe, which allows Superset startup to succeed and not be killed. See https://github.com/stackabletech/superset-operator/pull/654[superset-operator#654]. + +===== Open Policy Agent + +* *Breaking:* The per-rolegroup Services now only expose the HTTP port and contain a `-headless` suffix to better indicate their purpose and to be consistent with other operators. + See https://github.com/stackabletech/opa-operator/pull/748[opa-operator#748]. +* Previously the opa-operator ignored `envOverrides` set on role or rolegroup level. + With this release, the `envOverrides` are now properly propagated by the operator. + See https://github.com/stackabletech/opa-operator/pull/754[opa-operator#754]. +* The xref:opa:usage-guide/user-info-fetcher.adoc[User Info Fetcher (UIF)] is no longer marked as experimental. + See https://github.com/stackabletech/opa-operator/issues/751[opa-operator#751]. From d55077e1dfd5ae75b946e7163a51d9fc78b6dc08 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:36:31 +0100 Subject: [PATCH 14/54] docs: Add commons-operator improvements --- modules/ROOT/partials/release-notes/release-25.11.adoc | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index c9f70a37f..f8945a328 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -181,3 +181,10 @@ See https://github.com/stackabletech/superset-operator/pull/654[superset-operato See https://github.com/stackabletech/opa-operator/pull/754[opa-operator#754]. * The xref:opa:usage-guide/user-info-fetcher.adoc[User Info Fetcher (UIF)] is no longer marked as experimental. See https://github.com/stackabletech/opa-operator/issues/751[opa-operator#751]. + +===== Stackable commons-operator + +Reduce severity of Pod eviction errors. +Previously, the operator would produce lot's of `Cannot evict pod as it would violate the pod's disruption budget` errors. +With this fix, the error is reduced to an info instead. +See https://github.com/stackabletech/commons-operator/pull/372[commons-operator#372]. From 20772ef8b2e7d40195d7fc2fa87dcefca9b9b9a0 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:36:55 +0100 Subject: [PATCH 15/54] docs: Add listener-operator improvements --- .../partials/release-notes/release-25.11.adoc | 145 ++++++++++++++++++ 1 file changed, 145 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index f8945a328..3f12df067 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -188,3 +188,148 @@ Reduce severity of Pod eviction errors. Previously, the operator would produce lot's of `Cannot evict pod as it would violate the pod's disruption budget` errors. With this fix, the error is reduced to an info instead. See https://github.com/stackabletech/commons-operator/pull/372[commons-operator#372]. + +===== Stackable listener-operator + +* *Breaking:* Default ListenerClass `.spec.externalTrafficPolicy` to `null` to improve LoadBalancer support across various Kubernetes environments. + See https://github.com/stackabletech/listener-operator/pull/347[listener-operator#347]. +* *Breaking:* The listener-operator default preset changed from `stable-nodes` to `ephemeral-nodes`. + Previously, `external-stable` NodePorts pinned the Pod to a specific node, which caused problems with node rotations. +// TODO: See if we use code block here or not + Node pinning can be enabled via ListenerClass' new xref:listener-operator:listenerclass.adoc##_node_stickiness[`.spec.pinnedNodePorts`] field. + The preset can be configured using the following two commands: ++ +-- +[source,shell] +---- +helm --set preset=stable-nodes # or ephemeral-nodes or none +---- + +[source,shell] +---- +stackablectl --listener-class-preset stable-nodes # or ephemeral-nodes or none <1> +---- + +<1> It should be noted that `stackablectl` automatically detects k3s and kind clusters and uses the `stable-nodes` preset. + +See the tracking issue https://github.com/stackabletech/issues/issues/770[issues#770] for more details. +-- + +* *Breaking:* Helm values have changed to allow for separate configuration of affinity, resource, etc... between the operator deployment and the CSI driver pods. ++ +-- +// TODO: Confirm these are the final values for 25.11.0 +Container resources for the CSI Controller Service (`sdp/listener-operator` in the Deployment): + +[source,yaml] +---- +# Before +controller: + resources: + +# After +csiProvisioner: + controllerService: + resources: +---- + +Container image/resources for the external-provisioner (`sig-storage/csi-provisioner` in the Deployment): + +[source,yaml] +---- +# Before +csiProvisioner: + image: + resources: + +# After +csiProvisioner: + externalProvisioner: + image: + resources: +---- + +Container resources for the CSI Node Service (`sdp/listener-operator` in the DaemonSet): + +[source,yaml] +---- +# Before +node: + driver: + resources: + +# After +csiNodeDriver: + nodeService: + resources: +---- + +Container image/resources for the node-driver-registrar (`sig-storage/csi-node-driver-registrar` in the DaemonSet): + +[source,yaml] +---- +# Before +csiNodeDriverRegistrar: + image: + resources: + +# After +csiNodeDriver: + nodeDriverRegistrar: + image: + resources: +---- + +Settings that are now split: + +[source,yaml] +---- +# Before +podAnnotations: +podSecurityContext: +securityContext: +nodeSelector: +tolerations: +affinity: + +# After +csiProvisioner: + podAnnotations: + podSecurityContext: + nodeSelector: + tolerations: + affinity: + + controllerService: + securityContext: + +csiNodeDriver: + podAnnotations: + podSecurityContext: + nodeSelector: + tolerations: + affinity: + + nodeService: + securityContext: +---- + +See the tracking issue https://github.com/stackabletech/issues/issues/763[issues#763] and https://github.com/stackabletech/listener-operator/pull/334[listener-operator#334] for more details. +-- + +* As part of the Helm value changes listed above, some resource names have also been updated. ++ +[WARNING] +==== +It should be noted that generally no action is required, but that depends on whether or not your deployment scripts (eg: Kustomize) or monitoring/alerting system depends on any of the names and values. +==== +** Deployment `testing-listener-operator-deployment` has been renamed to `testing-listener-operator-csi-provisioner` +*** The `app.kubernetes.io/role` label value has changed from `controller` to `provisioner` +*** Container `listener-operator` has been renamed to `csi-controller-service` +** DaemonSet `listener-operator-node-daemonset` has been renamed to `listener-operator-csi-node-driver` +*** The `app.kubernetes.io/role` label value has changed from `node` to `node-driver` +*** Container `listener-operator` has been renamed to `csi-node-service` + + ++ +See https://github.com/stackabletech/listener-operator/pull/334[listener-operator#334] for more details. From 37d8cadf6272a3efb35d092caa2805b022538202 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:37:21 +0100 Subject: [PATCH 16/54] docs: Add secret-operator improvements --- .../partials/release-notes/release-25.11.adoc | 42 +++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 3f12df067..2c0800e18 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -333,3 +333,45 @@ It should be noted that generally no action is required, but that depends on whe + See https://github.com/stackabletech/listener-operator/pull/334[listener-operator#334] for more details. + +===== Stackable secret-operator + +* *Breaking:* The Helm Chart now deploys the secret-operator as two parts. + This separation is needed for CRD versioning and conversion by the operator. +** The controller (which reconciles resources, maintains CRDs and provides the CRD conversion webhook) runs as a Deployment with a single replica. +** The CSI server runs on every Kubernetes cluster node via a DaemonSet (this behaviour is unchanged). +** The Helm values are adjusted in accordance to the changes above. +// TODO: Should we list the changes explicitly like we did for listener-operator above? + See the secret-operator https://github.com/stackabletech/secret-operator/blob/25.11.0/CHANGELOG.md[changelog] for a complete overview of these changes. +** The old DaemonSet `secret-operator-daemonset` should be removed manually. + ++ +See https://github.com/stackabletech/secret-operator/pull/645[secret-operator#645]. + +* *Breaking:* The Stackable secret-operator no longer publishes retired and expired CA certificates: +** CA certificates are by default retired one hour before they expire. + This duration can be configured via `autoTls.ca.caCertificateRetirementDuration`. +** Expired and retired CA certificates are no longer published in Volumes and TrustStore. + ++ +See the xref:secret-operator:secretclass.adoc#ca-rotation[SecretClass] and xref:secret-operator:truststore.adoc[TrustStore] documentation as well as https://github.com/stackabletech/secret-operator/pull/650[secret-operator#650]. + +* The custom `samAccountName` generation is no longer marked as experimental. + To make this possible, the secret-operator is the first Stackable operator which supports CRD versioning. +** In version `v1alpha2` of the SecretClass, the `experimentalGenerateSamAccountName` field was renamed to `generateSamAccountName`. + See the xref:secret-operator:secretclass.adoc[SecretClass reference] for more details. +** The stored version of SecretClass is `v1alpha2`. + It is however still possible to apply and retrieve SecretClasses in `v1alpha1`. + The resources are automatically converted by the operator. +** The operator now deploys the CRDs for SecretClass and TrustStore by itself instead of relying on the Helm chart. + This enables the operator to automatically rotate and update the TLS certificate (`caBundle`) used for the conversion webhook. + ++ +See https://github.com/stackabletech/secret-operator/pull/634[secret-operator#634]. + +* The `certManager` backend is no longer marked as experimental. + In version `v1alpha2` of the SecretClass, the `experimentalCertManager` field was renamed to `certManager`. + See the xref:secret-operator:secretclass.adoc[SecretClass reference] for more details. + +* The operator now supports exporting the TrustStore CA certificate information to Secrets (in addition to ConfigMaps). + See https://github.com/stackabletech/secret-operator/pull/597[secret-operator#597]. From 0f5937a45e8b9778d09843ec337a833c9438af22 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:37:54 +0100 Subject: [PATCH 17/54] docs: Add new product LTS versions --- .../ROOT/partials/release-notes/release-25.11.adoc | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 2c0800e18..6710d1372 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -375,3 +375,17 @@ See https://github.com/stackabletech/secret-operator/pull/634[secret-operator#63 * The operator now supports exporting the TrustStore CA certificate information to Secrets (in addition to ConfigMaps). See https://github.com/stackabletech/secret-operator/pull/597[secret-operator#597]. + +==== Supported versions + +===== Product versions + +As with previous SDP releases, many product images have been updated to their latest versions. +Refer to the xref:operators:supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. + +====== New LTS versions + +The following product versions were already available before but are now marked as the xref:compliance:policies.adoc#_product_lifecycle_policy[LTS] version: + +* Apache Hive: https://github.com/stackabletech/docker-images/issues/1271[4.0.1 (LTS)] +* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1272[3.9.1 (LTS)] From d72daaa32732c6e2a916a2f4a52ed79ba567da64 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:38:20 +0100 Subject: [PATCH 18/54] docs: Add new product versions --- .../partials/release-notes/release-25.11.adoc | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 6710d1372..6bbf9cfb7 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -389,3 +389,22 @@ The following product versions were already available before but are now marked * Apache Hive: https://github.com/stackabletech/docker-images/issues/1271[4.0.1 (LTS)] * Apache Kafka: https://github.com/stackabletech/docker-images/issues/1272[3.9.1 (LTS)] + +====== New versions + +The following new product versions are now supported: + +* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1268[3.0.6 (LTS)] +* Apache Druid: https://github.com/stackabletech/docker-images/issues/1234[34.0.0] +* Apache HBase: https://github.com/stackabletech/docker-images/issues/1269[2.6.3 (LTS)] +* Apache Hadoop: https://github.com/stackabletech/docker-images/issues/1270[3.4.2 (LTS)] +* Apache Hive: https://github.com/stackabletech/docker-images/issues/1235[4.1.0] +* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1236[4.1.o (experimental)] +* Apache NiFi: https://github.com/stackabletech/docker-images/issues/1237[2.6.0 (LTS)] +* Apache Spark: https://github.com/stackabletech/docker-images/issues/1273[3.5.7 (LTS)], https://github.com/stackabletech/docker-images/issues/1273[4.0.1 (experimental)] +* Apache Superset: https://github.com/stackabletech/docker-images/issues/1274[4.1.4 (LTS)] +* Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1275[3.9.4 (LTS)] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1239[3.1.0 (LTS)] +* Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1238[1.8.0] +* Trino: https://github.com/stackabletech/docker-images/issues/1242[477 (LTS)] +* Vector: https://github.com/stackabletech/docker-images/issues/1233[0.49.0] From 7ae82304642078582b6cee34971b1c78bff34341 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:38:36 +0100 Subject: [PATCH 19/54] docs: Add deprecated product versions --- .../partials/release-notes/release-25.11.adoc | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 6bbf9cfb7..aef53e221 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -408,3 +408,20 @@ The following new product versions are now supported: * Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1238[1.8.0] * Trino: https://github.com/stackabletech/docker-images/issues/1242[477 (LTS)] * Vector: https://github.com/stackabletech/docker-images/issues/1233[0.49.0] + +====== Deprecated versions + +The following product versions are deprecated and will be removed in a later release: + +* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1266[2.9.3], https://github.com/stackabletech/docker-images/issues/1268[2.10.5] +* Apache Druid: https://github.com/stackabletech/docker-images/issues/1234[33.0.0] +* Apache HBase: https://github.com/stackabletech/docker-images/issues/1269[2.6.4] +* Apache Hadoop: https://github.com/stackabletech/docker-images/issues/1270[3.4.1] +* Apache Hive: https://github.com/stackabletech/docker-images/issues/1271[4.0.0] +* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1236[3.7.2] +* Apache NiFi: https://github.com/stackabletech/docker-images/issues/1237[1.27.0], https://github.com/stackabletech/docker-images/issues/1237[1.28.1], https://github.com/stackabletech/docker-images/issues/1237[2.4.0] +* Apache Spark: https://github.com/stackabletech/docker-images/issues/1273[3.5.6] +* Apache Superset: https://github.com/stackabletech/docker-images/issues/1241[4.0.2], https://github.com/stackabletech/docker-images/issues/1274[4.1.2] +* Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1275[3.9.3] +* Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1238[1.4.2] +* Trino: https://github.com/stackabletech/docker-images/issues/1242[451], https://github.com/stackabletech/docker-images/issues/1242[476] From 10bdc5bf25a24447858a164a1ce5a0b2aadd4792 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:38:53 +0100 Subject: [PATCH 20/54] docs: Add removed product versions --- .../partials/release-notes/release-25.11.adoc | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index aef53e221..aeb571448 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -425,3 +425,19 @@ The following product versions are deprecated and will be removed in a later rel * Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1275[3.9.3] * Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1238[1.4.2] * Trino: https://github.com/stackabletech/docker-images/issues/1242[451], https://github.com/stackabletech/docker-images/issues/1242[476] + +====== Removed versions + +The following product versions are no longer supported. +These images for released product versions remain available https://oci.stackable.tech/[here,window=_blank]. +Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here,window=_blank]. + +* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1268[2.10.4] +* Apache Druid: https://github.com/stackabletech/docker-images/issues/1234[31.0.1] +* Apache HBase: https://github.com/stackabletech/docker-images/issues/1269[2.6.1] +* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1272[3.9.0] +* Apache Spark: https://github.com/stackabletech/docker-images/issues/1273[3.5.5], https://github.com/stackabletech/docker-images/issues/1273[4.0.0] +* Apache Superset: https://github.com/stackabletech/docker-images/issues/1274[4.1.1] +* Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1238[1.0.1] +* Trino: https://github.com/stackabletech/docker-images/issues/1242[470] +* Vector: https://github.com/stackabletech/docker-images/issues/1233[0.47.0] From b285d83641f46a5d591d3120c6e52037be0c86f4 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:39:16 +0100 Subject: [PATCH 21/54] docs: Add supported Kubernetes versions --- .../ROOT/partials/release-notes/release-25.11.adoc | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index aeb571448..82cfac238 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -441,3 +441,16 @@ Information on how to browse the registry can be found xref:contributor:project- * Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1238[1.0.1] * Trino: https://github.com/stackabletech/docker-images/issues/1242[470] * Vector: https://github.com/stackabletech/docker-images/issues/1233[0.47.0] + +===== Kubernetes versions + +This release supports the following Kubernetes versions: + +* `1.34` +* `1.33` +* `1.32` +* `1.31` + +These Kubernetes versions are no longer supported: + +* `1.30` From 3b0aa357d95ee8c0592ed8222731701c78ad2e24 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:39:33 +0100 Subject: [PATCH 22/54] docs: Add supported OpenShift versions --- .../ROOT/partials/release-notes/release-25.11.adoc | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 82cfac238..8f7b26010 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -454,3 +454,16 @@ This release supports the following Kubernetes versions: These Kubernetes versions are no longer supported: * `1.30` + +===== OpenShift versions + +This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions: + +* `4.20` +* `4.19` +* `4.18` + +These OpenShift versions are no longer supported: + +* `4.17` +* `4.16` From 3f9abed2fc868d3b3d9bb300b76f2c05f3d2871d Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:40:21 +0100 Subject: [PATCH 23/54] docs: Add upgrade instructions --- .../partials/release-notes/release-25.11.adoc | 141 ++++++++++++++++++ 1 file changed, 141 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 8f7b26010..01e6b3115 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -467,3 +467,144 @@ These OpenShift versions are no longer supported: * `4.17` * `4.16` + +==== Upgrade from 25.7 + +===== Using stackablectl + +====== Upgrade with a single command + +Starting with stackablectl Release 1.0.0 the multiple consecutive commands described below can be shortened to just one command, which executes exactly those steps on its own. + +[source,console] +---- +$ stackablectl release upgrade 25.11 +---- + +====== Upgrade with multiple consecutive commands + +Uninstall the `25.7` release + +[source,console] +---- +$ stackablectl release uninstall 25.7 + +Uninstalled release '25.7' + +Use "stackablectl release list" to list available releases. +# ... +---- + +Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. +The reason for this is that helm will uninstall the operators but not the CRDs. +This can be done using `kubectl replace`. + +// TODO: Mention that the CRDs of the secret-operator don't need to be replaced manually. + +[source,shell] +---- +kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/25.11.0/deploy/helm/airflow-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/25.11.0/deploy/helm/commons-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/25.11.0/deploy/helm/druid-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/25.11.0/deploy/helm/hbase-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/25.11.0/deploy/helm/hdfs-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/25.11.0/deploy/helm/hive-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/25.11.0/deploy/helm/kafka-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/25.11.0/deploy/helm/listener-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/25.11.0/deploy/helm/nifi-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/25.11.0/deploy/helm/opa-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/25.11.0/deploy/helm/spark-k8s-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/25.11.0/deploy/helm/superset-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/25.11.0/deploy/helm/trino-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/25.11.0/deploy/helm/zookeeper-operator/crds/crds.yaml +---- + +[source] +---- +customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced +... +---- + +Install the `25.11` release + +[source,console] +---- +$ stackablectl release install 25.11 + +Installed release '25.11' + +Use "stackablectl operator installed" to list installed operators. +---- + +===== Using Helm + +Use `helm list` to list the currently installed operators. + +You can use the following command to uninstall all operators that are part of the `25.7` release: + +[source,console] +---- +$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator +release "airflow-operator" uninstalled +release "commons-operator" uninstalled +... +---- + +Afterward you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. +The reason for this is that helm will uninstall the operators but not the CRDs. +This can be done using `kubectl replace`. + +// TODO: Mention that the CRDs of the secret-operator don't need to be replaced manually. + +[source,shell] +---- +kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/25.11.0/deploy/helm/airflow-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/25.11.0/deploy/helm/commons-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/25.11.0/deploy/helm/druid-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/25.11.0/deploy/helm/hbase-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/25.11.0/deploy/helm/hdfs-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/25.11.0/deploy/helm/hive-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/25.11.0/deploy/helm/kafka-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/25.11.0/deploy/helm/listener-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/25.11.0/deploy/helm/nifi-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/25.11.0/deploy/helm/opa-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/25.11.0/deploy/helm/spark-k8s-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/25.11.0/deploy/helm/superset-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/25.11.0/deploy/helm/trino-operator/crds/crds.yaml +kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/25.11.0/deploy/helm/zookeeper-operator/crds/crds.yaml +---- + +[source] +---- +customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced +customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced +... +---- + +Install the `25.11` release + +NOTE: `helm repo` subcommands are not supported for OCI registries. +The operators are installed directly, without adding the Helm Chart repository first. + +[source,console] +---- +helm install --wait airflow-operator oci://oci.stackable.tech/sdp-charts/airflow-operator --version 25.11.0 +helm install --wait commons-operator oci://oci.stackable.tech/sdp-charts/commons-operator --version 25.11.0 +helm install --wait druid-operator oci://oci.stackable.tech/sdp-charts/druid-operator --version 25.11.0 +helm install --wait hbase-operator oci://oci.stackable.tech/sdp-charts/hbase-operator --version 25.11.0 +helm install --wait hdfs-operator oci://oci.stackable.tech/sdp-charts/hdfs-operator --version 25.11.0 +helm install --wait hive-operator oci://oci.stackable.tech/sdp-charts/hive-operator --version 25.11.0 +helm install --wait kafka-operator oci://oci.stackable.tech/sdp-charts/kafka-operator --version 25.11.0 +helm install --wait listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --version 25.11.0 +helm install --wait nifi-operator oci://oci.stackable.tech/sdp-charts/nifi-operator --version 25.11.0 +helm install --wait opa-operator oci://oci.stackable.tech/sdp-charts/opa-operator --version 25.11.0 +helm install --wait secret-operator oci://oci.stackable.tech/sdp-charts/secret-operator --version 25.11.0 +helm install --wait spark-k8s-operator oci://oci.stackable.tech/sdp-charts/spark-k8s-operator --version 25.11.0 +helm install --wait superset-operator oci://oci.stackable.tech/sdp-charts/superset-operator --version 25.11.0 +helm install --wait trino-operator oci://oci.stackable.tech/sdp-charts/trino-operator --version 25.11.0 +helm install --wait zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version 25.11.0 +---- + From 5ff43c3a800665011bbba7f44a30e08a67c2fa54 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:41:27 +0100 Subject: [PATCH 24/54] docs: Add release date --- modules/ROOT/partials/release-notes/release-25.11.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 01e6b3115..581ae0366 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -2,7 +2,7 @@ === 25.11.0 -Released on YYYY-MM-DD. +Released on 2025-11-07. (Optional description / introduction) ==== New platform features From 282cc3f59183edd5253dc739ab3a0e87f376be51 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 16:43:29 +0100 Subject: [PATCH 25/54] chore: Remove trailing empty line --- modules/ROOT/partials/release-notes/release-25.11.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 581ae0366..c4dfe2f12 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -607,4 +607,3 @@ helm install --wait superset-operator oci://oci.stackable.tech/sdp-charts/supers helm install --wait trino-operator oci://oci.stackable.tech/sdp-charts/trino-operator --version 25.11.0 helm install --wait zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version 25.11.0 ---- - From 07aae0bbaaea2b16e8febf4502e1ffd933385b61 Mon Sep 17 00:00:00 2001 From: Techassi Date: Tue, 11 Nov 2025 17:04:06 +0100 Subject: [PATCH 26/54] docs: Add EoS check feature --- .../partials/release-notes/release-25.11.adoc | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index c4dfe2f12..41e12e825 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -21,6 +21,24 @@ Traffic between Open Policy Agent (OPA) and clients can be encrypted using TLS b The authorizers for Trino and NiFi automatically integrate with these secured OPA deployments and verify the authenticity of the server certificates when TLS for OPA is enabled. See the xref:opa:usage-guide/tls.adoc[TLS encryption documentation page] and https://github.com/stackabletech/opa-operator/issues/581[opa-operator#581]. +====== End-of-Support (EoS) warning + +All operators now emit a warning message on startup and in a regular interval when it may have reached end-of-support. +Most of our operators reach end-of-support one year after they have been released which roughly translates to three SDP releases. +This is in accordance with our xref:compliance:policies.adoc[support policy]. +The interval can be adjusted or the check can be disabled completely via Helm values. + +[source,yaml] +---- +maintenance: + endOfSupportCheck: + enabled: true + mode: offline # only offline is currently supported + interval: 24h # A human-readable duration +---- + +See https://github.com/stackabletech/issues/issues/733[issues#733]. + ====== Miscellaneous // TODO: Do we want to include this? From f3514f9e2f2dd37b9e49e84f65919497bf4f5e51 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 11:20:48 +0100 Subject: [PATCH 27/54] docs: Introduce a new "platform fixes" section --- .../partials/release-notes/release-25.11.adoc | 58 +++++++++++-------- 1 file changed, 33 insertions(+), 25 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 41e12e825..3be5453b6 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -9,12 +9,6 @@ Released on 2025-11-07. ===== General -====== Custom image selection - -Previously, when using custom images in combination with a SHA digest like `oci.stackable.tech/sdp/spark-k8s@sha256:c8b7...`, all operators created invalid labels `app.kubernetes.io/version` for their applied resources. -This was fixed by checking and replacing invalid characters in the created labels when a SHA digest is used to select the custom image. -See https://github.com/stackabletech/operator-rs/pull/1076[operator-rs#1076]. - ====== Security Traffic between Open Policy Agent (OPA) and clients can be encrypted using TLS by enabling it in the OPA custom resource. @@ -146,16 +140,6 @@ priorityClassName: ... ===== Apache Airflow -* Previously, a missing OPA ConfigMap would crash the operator. - With this release, we don't panic on an invalid authorization config. - See https://github.com/stackabletech/airflow-operator/pull/667[airflow-operator#667]. -* Previously, OPA authorization for Airflow 3 was not working. - With this release, the operator now sets the required environment variables. - See https://github.com/stackabletech/airflow-operator/pull/668[airflow-operator#668]. -* Allow multiple Airflows in the same namespace to use Kubernetes executors. - Previously, the operator would always use the same name for the executor Pod template ConfigMap. - Thus when deploying multiple Airflow instances in the same namespace, the ConfigMaps would conflict. - See https://github.com/stackabletech/airflow-operator/pull/678[airflow-operator#678]. // TODO: Is there only ONE key or should we clarify WHICH key this is? * The JWT key is now created by the the operator internally. The same applies to the key previously defined in the credentials secret under `connections.secretKey`: this change is non-breaking, as `connections.secretKey` will be ignored if supplied. @@ -178,12 +162,6 @@ Only use this setting if you know what you are doing! The Apache NiFi xref:nifi:usage_guide/monitoring.adoc#_configure_metrics_in_nifi_2_x_x[monitoring documentation] page has been updated to include guidance on how to scrape NiFi 2.X.X metrics using mTLS. See https://github.com/stackabletech/nifi-operator/issues/813[nifi-operator#813]. -===== Apache Spark - -Spark Connect: Previously the property `spec.image.pullSecrets` was ignored by the operator when creating the executor templates. -This has now been corrected in the operator code. -See https://github.com/stackabletech/spark-k8s-operator/issues/600[spark-k8s-operator#600]. - ===== Apache Superset Previously, there was a chance containers would not start, because Superset was starting too slowly and was killed because of a failing liveness probe. @@ -194,9 +172,6 @@ See https://github.com/stackabletech/superset-operator/pull/654[superset-operato * *Breaking:* The per-rolegroup Services now only expose the HTTP port and contain a `-headless` suffix to better indicate their purpose and to be consistent with other operators. See https://github.com/stackabletech/opa-operator/pull/748[opa-operator#748]. -* Previously the opa-operator ignored `envOverrides` set on role or rolegroup level. - With this release, the `envOverrides` are now properly propagated by the operator. - See https://github.com/stackabletech/opa-operator/pull/754[opa-operator#754]. * The xref:opa:usage-guide/user-info-fetcher.adoc[User Info Fetcher (UIF)] is no longer marked as experimental. See https://github.com/stackabletech/opa-operator/issues/751[opa-operator#751]. @@ -394,6 +369,39 @@ See https://github.com/stackabletech/secret-operator/pull/634[secret-operator#63 * The operator now supports exporting the TrustStore CA certificate information to Secrets (in addition to ConfigMaps). See https://github.com/stackabletech/secret-operator/pull/597[secret-operator#597]. +==== Platform fixes + +===== Custom image selection + +Previously, when using custom images in combination with a SHA digest like `oci.stackable.tech/sdp/spark-k8s@sha256:c8b7...`, all operators created invalid labels `app.kubernetes.io/version` for their applied resources. +This was fixed by checking and replacing invalid characters in the created labels when a SHA digest is used to select the custom image. +See https://github.com/stackabletech/operator-rs/pull/1076[operator-rs#1076]. + +===== Apache Airflow + +* Previously, a missing OPA ConfigMap would crash the operator. + With this release, we don't panic on an invalid authorization config. + See https://github.com/stackabletech/airflow-operator/pull/667[airflow-operator#667]. +* Previously, OPA authorization for Airflow 3 was not working. + With this release, the operator now sets the required environment variables. + See https://github.com/stackabletech/airflow-operator/pull/668[airflow-operator#668]. +* Allow multiple Airflows in the same namespace to use Kubernetes executors. + Previously, the operator would always use the same name for the executor Pod template ConfigMap. + Thus when deploying multiple Airflow instances in the same namespace, the ConfigMaps would conflict. + See https://github.com/stackabletech/airflow-operator/pull/678[airflow-operator#678]. + +===== Apache Spark + +Spark Connect: Previously the property `spec.image.pullSecrets` was ignored by the operator when creating the executor templates. +This has now been corrected in the operator code. +See https://github.com/stackabletech/spark-k8s-operator/issues/600[spark-k8s-operator#600]. + +===== Open Policy Agent + +Previously the opa-operator ignored `envOverrides` set on role or rolegroup level. +With this release, the `envOverrides` are now properly propagated by the operator. +See https://github.com/stackabletech/opa-operator/pull/754[opa-operator#754]. + ==== Supported versions ===== Product versions From 4e692a1a7437e7e0c20edd36d33dfc1dac6cfde8 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 11:21:26 +0100 Subject: [PATCH 28/54] docs: Add initial OpenSearch announcement --- modules/ROOT/partials/release-notes/release-25.11.adoc | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 3be5453b6..31010a310 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -71,7 +71,11 @@ This release adds a dedicated per-rolegroup `-metrics` Service, which can be use // TODO: How do we expose more metrics? The PR this snippet comes from doesn't seem to expose more metrics. // The Expose more Prometheus metrics, such as successful or failed bundle loads and information about the OPA environment -====== Trino +===== OpenSearch + +As of this release OpenSearch is part of the SDP. +We initially support version link:#new-product-versions-25_11_0[3.1.0], which is also marked as the LTS line going forward. + * The operator now supports configuring fault-tolerant execution for Trino via the TrinoCluster CRD. See the xref:trino:usage-guide/fault-tolerant-execution.adoc[documentation page] and https://github.com/stackabletech/trino-operator/pull/779[trino-operator#779]. @@ -416,6 +420,7 @@ The following product versions were already available before but are now marked * Apache Hive: https://github.com/stackabletech/docker-images/issues/1271[4.0.1 (LTS)] * Apache Kafka: https://github.com/stackabletech/docker-images/issues/1272[3.9.1 (LTS)] +[#new-product-versions-25_11_0] ====== New versions The following new product versions are now supported: From 5bdab9ca0b31fbcd06172672951404813168291b Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 11:21:50 +0100 Subject: [PATCH 29/54] docs: Use correct heading level --- modules/ROOT/partials/release-notes/release-25.11.adoc | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 31010a310..65108542f 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -65,7 +65,7 @@ See https://github.com/stackabletech/nifi-operator/issues/812[nifi-operator#812] The support is marked as experimental because Spark 4.0.0 has known compatibility issues with https://github.com/apache/hbase-connectors/pull/130[Apache HBase] and https://github.com/apache/iceberg/issues/13358[Apache Iceberg]. See https://github.com/stackabletech/spark-k8s-operator/issues/586[spark-k8s-operator#586]. -====== Open Policy Agent +===== Open Policy Agent This release adds a dedicated per-rolegroup `-metrics` Service, which can be used to scrape Prometheus metrics. // TODO: How do we expose more metrics? The PR this snippet comes from doesn't seem to expose more metrics. @@ -76,6 +76,7 @@ This release adds a dedicated per-rolegroup `-metrics` Service, which can be use As of this release OpenSearch is part of the SDP. We initially support version link:#new-product-versions-25_11_0[3.1.0], which is also marked as the LTS line going forward. +===== Trino * The operator now supports configuring fault-tolerant execution for Trino via the TrinoCluster CRD. See the xref:trino:usage-guide/fault-tolerant-execution.adoc[documentation page] and https://github.com/stackabletech/trino-operator/pull/779[trino-operator#779]. From a2ded398cf7e530b9be9f9cf68658e666b1e4b84 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 12:04:59 +0100 Subject: [PATCH 30/54] docs: Add more content to the OpenSearch section --- .../ROOT/partials/release-notes/release-25.11.adoc | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 65108542f..8371f2ce2 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -73,9 +73,21 @@ This release adds a dedicated per-rolegroup `-metrics` Service, which can be use ===== OpenSearch -As of this release OpenSearch is part of the SDP. +The Stackable platform now provides an operator for OpenSearch. We initially support version link:#new-product-versions-25_11_0[3.1.0], which is also marked as the LTS line going forward. +OpenSearch is a powerful search and analytics engine built on Apache Lucene. +OpenSearch clusters can be specified in custom resources similar to other Stackable operators. +For instance, a cluster with OpenSearch nodes of different types and replication factors can be defined. +Logging, Monitoring and service exposition with ListenerClasses is supported as well. +As the operator is still in an early development phase, special care was taken to allow extensive overriding with xref:concepts:overrides.adoc#config-overrides[`configOverrides`] and xref:concepts:overrides.adoc#pod-overrides[`podOverrides`]. + +The operator only manages the OpenSearch back-end. +The OpenSearch Dashboards front-end can be installed via the https://github.com/opensearch-project/helm-charts/tree/opensearch-dashboards-3.1.0/charts/opensearch-dashboards[official Helm chart]. +Stackable provides a supported image for OpenSearch Dashboard which can be used with this Helm chart. + +See the xref:opensearch:index.adoc[OpenSearch documentation] page for more details. + ===== Trino * The operator now supports configuring fault-tolerant execution for Trino via the TrinoCluster CRD. From c62e23f366b5ba092c7e29e58d149926763510e9 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 12:43:07 +0100 Subject: [PATCH 31/54] docs: Explicitly mention secret-operator Helm changes --- .../partials/release-notes/release-25.11.adoc | 108 +++++++++++++++++- 1 file changed, 104 insertions(+), 4 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 8371f2ce2..079917924 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -138,7 +138,13 @@ csiProvisioner: csiNodeDriver: priorityClassName: ... -# TODO: List/check secret-operator here as well +# Secret operator +controllerService: + priorityClassName: ... + +csiNodeDriver: + priorityClassName: ... + # All other operators priorityClassName: ... ---- @@ -350,10 +356,104 @@ See https://github.com/stackabletech/listener-operator/pull/334[listener-operato This separation is needed for CRD versioning and conversion by the operator. ** The controller (which reconciles resources, maintains CRDs and provides the CRD conversion webhook) runs as a Deployment with a single replica. ** The CSI server runs on every Kubernetes cluster node via a DaemonSet (this behaviour is unchanged). -** The Helm values are adjusted in accordance to the changes above. -// TODO: Should we list the changes explicitly like we did for listener-operator above? - See the secret-operator https://github.com/stackabletech/secret-operator/blob/25.11.0/CHANGELOG.md[changelog] for a complete overview of these changes. ** The old DaemonSet `secret-operator-daemonset` should be removed manually. +** The Helm values are adjusted in accordance to the changes above. ++ +-- +Both the external provisioner and the node driver registrar have been moved under `csiNodeDriver`: + +[source,yaml] +---- +# Before +csiProvisioner: + resources: + +csiNodeDriverRegistrar: + resources: + +# After +csiNodeDriver: + externalProvisioner: + resources: + nodeDriverRegistrar: + resources: +---- + +The secret-operator is now deployed through a Deployment and a DaemonSet. +As such, the resources of both secret-operator instances can be controlled separately: + +[source,yaml] +---- +# Before +node: + driver: + resources: + +# After +csiNodeDriver: + nodeService: + resources: + +controllerService: + resources: +---- + +The `securityContext` has been split into two parts: + +[source,yaml] +---- +# Before +securityContext: + +# After +csiNodeDriver: + nodeService: + securityContext: + +controllerService: + securityContext: +---- + +Settings that are now split: + +[source,yaml] +---- +# Before +podAnnotations: +podSecurityContext: +nodeSelector: +tolerations: +affinity: + +# After +csiNodeDriver: + podAnnotations: + podSecurityContext: + nodeSelector: + tolerations: + affinity: + +controllerService: + podAnnotations: + podSecurityContext: + nodeSelector: + tolerations: + affinity: +---- + +Settings that have moved: + +[source,yaml] +---- +# Before +kubeletDir: + +# After +csiNodeDriver: + kubeletDir: +---- + +-- + See https://github.com/stackabletech/secret-operator/pull/645[secret-operator#645]. From b04f768d02f205b111c2d0a4bc886b14a8d69022 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 13:00:11 +0100 Subject: [PATCH 32/54] docs: Mention increased metrics exposure for OPA --- modules/ROOT/partials/release-notes/release-25.11.adoc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 079917924..8de1920f7 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -68,8 +68,7 @@ See https://github.com/stackabletech/nifi-operator/issues/812[nifi-operator#812] ===== Open Policy Agent This release adds a dedicated per-rolegroup `-metrics` Service, which can be used to scrape Prometheus metrics. -// TODO: How do we expose more metrics? The PR this snippet comes from doesn't seem to expose more metrics. -// The Expose more Prometheus metrics, such as successful or failed bundle loads and information about the OPA environment +Additionally, the operator exposes more Prometheus metrics, such as successful or failed bundle loads and information about the OPA environment. ===== OpenSearch From 135e877a405d841fad738a0bc9628849f889a940 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 15:02:24 +0100 Subject: [PATCH 33/54] docs: Add Kafka KRaft section --- modules/ROOT/partials/release-notes/release-25.11.adoc | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 8de1920f7..fdde66d60 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -48,6 +48,13 @@ See https://github.com/stackabletech/demos/pull/289[demos#289]. * The xref:demos:airflow-scheduled-job.adoc[`airflow-scheduled-job`] demo for Airflow has been extended to showcase some of the new Airflow 3.x features in the context of SDP i.e. event scheduling (with Kafka), triggerer actions and user authorization with OPA and the SDP OPA authorizer. See https://github.com/stackabletech/demos/issues/223[demos#223]. +===== Apache Kafka + +This release adds experimental support for KRaft managed Kafka clusters. +KRaft Controllers can be deployed instead of Apache ZooKeeper to manage the state of Kafka. +KRaft is supported by all Kafka versions provided by SDP, and starting with Kafka 4.X it is the only cluster management version available. +See https://github.com/stackabletech/kafka-operator/pull/889[kafka-operator#889]. + ===== Apache NiFi A patch was added which allows disabling the SNI (Server Name Indication) checks for NiFi. From af3de69b0404264f4e693cb37b25d57dcdcd2722 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 15:02:59 +0100 Subject: [PATCH 34/54] docs: Improve/expand metrics section --- .../partials/release-notes/release-25.11.adoc | 38 +++++++++++++++---- 1 file changed, 31 insertions(+), 7 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index fdde66d60..48f888e42 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -117,14 +117,38 @@ This includes 2 critical and 18 high-severity CVEs. This release includes various improvements in regards to metrics collection and exposition. Previously, some operators did not expose Prometheus annotations containing the HTTP(S) scheme or the metrics path and port. -These annotations are now available which allows custom relabel configs in Prometheus to scrape the metric endpoints. - -* Apache HBase: The `prometheus.io/scrape` label is now only available on the `metrics` Service (instead of the `headless` service), which uses `metrics` as the port name instead of the previous `ui-http`/`ui-https` port name. +These annotations are now available which allows custom relabel configs in Prometheus to scrape the metric endpoints: + +* Apache Airflow: https://github.com/stackabletech/airflow-operator/pull/698[airflow-operator#698]. +* Apache Druid: https://github.com/stackabletech/druid-operator/pull/761[airflow-operator#761]. +* Apache Hive: https://github.com/stackabletech/hive-operator/pull/641[hive-operator#641]. +* Apache Kafka: https://github.com/stackabletech/kafka-operator/pull/897[kafka-operator#897]. +* Apache NiFi: https://github.com/stackabletech/nifi-operator/pull/855[nifi-operator#855]. +* Apache Spark: https://github.com/stackabletech/spark-k8s-operator/pull/619[spark-k8s-operator#619]. +* Apache Superset: https://github.com/stackabletech/superset-operator/pull/671[superset-operator#671]. +* Apache ZooKeeper: https://github.com/stackabletech/zookeeper-operator/pull/978[zookeeper-operator#978]. +* Open Policy Agent: https://github.com/stackabletech/opa-operator/pull/767[opa-operator#767]. +* Trino: https://github.com/stackabletech/trino-operator/pull/807[trino-operator#807]. + +In addition to the annotation changes listed above, the following changes were made: + +* Apache HBase: The `prometheus.io/scrape` label is now only available on the `metrics` Service (instead of the `headless` Service), which uses `metrics` as the port name instead of the previous `ui-http`/`ui-https` port name. See https://github.com/stackabletech/hbase-operator/pull/701[hbase-operator#701]. -* Apache Airflow: The operator now adds the appropriate Prometheus annotations. - See https://github.com/stackabletech/airflow-operator/pull/698[airflow-operator#698]. -* Apache Druid: The operator now adds the appropriate Prometheus annotations. - See https://github.com/stackabletech/druid-operator/pull/761[airflow-operator#761]. +* Apache Hadoop: The `metrics` Service previously exposed the JMX metrics via the `metrics` port. + In this release, the JMX metrics have been moved to the `jmx-metrics` port. + The `metrics` port now instead exposes the native Prometheus metrics. ++ +-- +[WARNING] +==== +Care needs to be taken because the metrics format changed. +==== + +See https://github.com/stackabletech/hdfs-operator/pull/721[hdfs-operator#721]. +-- + +* Apache Kafka: Additionally, the `--` Service was replaced with `---headless` and `---metrics` Services. + See https://github.com/stackabletech/kafka-operator/pull/897[kafka-operator#897]. ====== Miscellaneous From d81f5edc583794099603ee1a1fb302ffa968a067 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 15:03:37 +0100 Subject: [PATCH 35/54] docs: Move Superset startup probe section --- .../ROOT/partials/release-notes/release-25.11.adoc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 48f888e42..c8f6a815c 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -215,12 +215,6 @@ Only use this setting if you know what you are doing! The Apache NiFi xref:nifi:usage_guide/monitoring.adoc#_configure_metrics_in_nifi_2_x_x[monitoring documentation] page has been updated to include guidance on how to scrape NiFi 2.X.X metrics using mTLS. See https://github.com/stackabletech/nifi-operator/issues/813[nifi-operator#813]. -===== Apache Superset - -Previously, there was a chance containers would not start, because Superset was starting too slowly and was killed because of a failing liveness probe. -Now, we add a proper startup probe, which allows Superset startup to succeed and not be killed. -See https://github.com/stackabletech/superset-operator/pull/654[superset-operator#654]. - ===== Open Policy Agent * *Breaking:* The per-rolegroup Services now only expose the HTTP port and contain a `-headless` suffix to better indicate their purpose and to be consistent with other operators. @@ -543,6 +537,12 @@ Spark Connect: Previously the property `spec.image.pullSecrets` was ignored by t This has now been corrected in the operator code. See https://github.com/stackabletech/spark-k8s-operator/issues/600[spark-k8s-operator#600]. +===== Apache Superset + +Previously, there was a chance containers would not start, because Superset was starting too slowly and was killed because of a failing liveness probe. +Now, we add a proper startup probe, which allows Superset startup to succeed and not be killed. +See https://github.com/stackabletech/superset-operator/pull/654[superset-operator#654]. + ===== Open Policy Agent Previously the opa-operator ignored `envOverrides` set on role or rolegroup level. From 28a8a28df7b882e824a8b4dac40c45568160d2b4 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 15:04:08 +0100 Subject: [PATCH 36/54] docs: Add release highlights --- .../ROOT/partials/release-notes/release-25.11.adoc | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index c8f6a815c..79992515e 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -3,7 +3,16 @@ === 25.11.0 Released on 2025-11-07. -(Optional description / introduction) + +[TIP,caption=Release highlights] +==== +* The Stackable platform now provides an operator for OpenSearch. +* All operators now correctly handle multiple CA certificates. +* The User Info Fetcher (UIF) is no longer marked as experimental. +* SecretClass `v1alpha2` is now available: +** The custom `samAccountName` generation is no longer marked as experimental. +** The `certManager` backend is no longer marked as experimental. +==== ==== New platform features From 407279ccbc41451027ddad9823ed7cae34278936 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 15:33:16 +0100 Subject: [PATCH 37/54] docs: Add various small tweaks --- .../partials/release-notes/release-25.11.adoc | 27 +++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 79992515e..27c42d866 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -92,36 +92,35 @@ The Stackable platform now provides an operator for OpenSearch. We initially support version link:#new-product-versions-25_11_0[3.1.0], which is also marked as the LTS line going forward. OpenSearch is a powerful search and analytics engine built on Apache Lucene. -OpenSearch clusters can be specified in custom resources similar to other Stackable operators. +OpenSearch clusters can be defined via custom resources similar to other Stackable operators. For instance, a cluster with OpenSearch nodes of different types and replication factors can be defined. Logging, Monitoring and service exposition with ListenerClasses is supported as well. As the operator is still in an early development phase, special care was taken to allow extensive overriding with xref:concepts:overrides.adoc#config-overrides[`configOverrides`] and xref:concepts:overrides.adoc#pod-overrides[`podOverrides`]. The operator only manages the OpenSearch back-end. The OpenSearch Dashboards front-end can be installed via the https://github.com/opensearch-project/helm-charts/tree/opensearch-dashboards-3.1.0/charts/opensearch-dashboards[official Helm chart]. -Stackable provides a supported image for OpenSearch Dashboard which can be used with this Helm chart. +Stackable provides a supported image for OpenSearch Dashboards which can be used with this Helm chart. See the xref:opensearch:index.adoc[OpenSearch documentation] page for more details. ===== Trino -* The operator now supports configuring fault-tolerant execution for Trino via the TrinoCluster CRD. +* The operator now supports configuring fault-tolerant execution via the TrinoCluster CRD. See the xref:trino:usage-guide/fault-tolerant-execution.adoc[documentation page] and https://github.com/stackabletech/trino-operator/pull/779[trino-operator#779]. * The Trino client spooling protocol can now be configured using the `spec.clusterConfig.clientProtocol.spooling` property. -// TODO: Is this an S3Connection (CR)? - Users can configure an S3 connection and the location or spooling segments. - Additional properties can be added using the `configOverrides` mechanism for the `spooling-manager.properties` file. + Users can configure an xref:concepts:s3.adoc[S3Connection] and the location of spooling segments. + Additional properties can be added using the xref:concepts:overrides.adoc#config-overrides[`configOverrides`] mechanism for the `spooling-manager.properties` file. See the xref:trino:usage-guide/client-spooling-protocol.adoc[client spooling protocol] documentation page and https://github.com/stackabletech/trino-operator/pull/793[trino-operator#793]. ==== Platform improvements -===== Vulnerabilities +===== General + +====== Vulnerabilities 37 CVEs were fixed in the Stackable product images. This includes 2 critical and 18 high-severity CVEs. -===== General - ====== Observability This release includes various improvements in regards to metrics collection and exposition. @@ -256,10 +255,10 @@ helm --set preset=stable-nodes # or ephemeral-nodes or none [source,shell] ---- -stackablectl --listener-class-preset stable-nodes # or ephemeral-nodes or none <1> +stackablectl --listener-class-preset stable-nodes # or ephemeral-nodes or none ---- -<1> It should be noted that `stackablectl` automatically detects k3s and kind clusters and uses the `stable-nodes` preset. +It should be noted that `stackablectl` automatically detects k3s and kind clusters and uses the `stable-nodes` preset since version xref:management:stackablectl:release-notes.adoc#_1_2_0[1.2.0]. See the tracking issue https://github.com/stackabletech/issues/issues/770[issues#770] for more details. -- @@ -549,7 +548,7 @@ See https://github.com/stackabletech/spark-k8s-operator/issues/600[spark-k8s-ope ===== Apache Superset Previously, there was a chance containers would not start, because Superset was starting too slowly and was killed because of a failing liveness probe. -Now, we add a proper startup probe, which allows Superset startup to succeed and not be killed. +This has now been fixed by adding a proper startup probe, which allows Superset startup to succeed and not be killed. See https://github.com/stackabletech/superset-operator/pull/654[superset-operator#654]. ===== Open Policy Agent @@ -582,7 +581,7 @@ The following new product versions are now supported: * Apache HBase: https://github.com/stackabletech/docker-images/issues/1269[2.6.3 (LTS)] * Apache Hadoop: https://github.com/stackabletech/docker-images/issues/1270[3.4.2 (LTS)] * Apache Hive: https://github.com/stackabletech/docker-images/issues/1235[4.1.0] -* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1236[4.1.o (experimental)] +* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1236[4.1.0 (experimental)] * Apache NiFi: https://github.com/stackabletech/docker-images/issues/1237[2.6.0 (LTS)] * Apache Spark: https://github.com/stackabletech/docker-images/issues/1273[3.5.7 (LTS)], https://github.com/stackabletech/docker-images/issues/1273[4.0.1 (experimental)] * Apache Superset: https://github.com/stackabletech/docker-images/issues/1274[4.1.4 (LTS)] @@ -657,7 +656,7 @@ These OpenShift versions are no longer supported: ====== Upgrade with a single command -Starting with stackablectl Release 1.0.0 the multiple consecutive commands described below can be shortened to just one command, which executes exactly those steps on its own. +Starting with `stackablectl` xref:management:stackablectl:release-notes.adoc#_1_0_0[1.0.0] the multiple consecutive commands described below can be shortened to just one command, which executes exactly those steps on its own. [source,console] ---- From 0fd5ae08900aeff474c9b544a0dcc4425afb1a0e Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 15:45:34 +0100 Subject: [PATCH 38/54] docs: Update release notes template --- .../release-notes/release-template.adoc | 152 +++++++++++++----- 1 file changed, 111 insertions(+), 41 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-template.adoc b/modules/ROOT/partials/release-notes/release-template.adoc index a0dc6bea6..6166d3d9f 100644 --- a/modules/ROOT/partials/release-notes/release-template.adoc +++ b/modules/ROOT/partials/release-notes/release-template.adoc @@ -8,74 +8,142 @@ Released on YYYY-MM-DD. (Optional description / introduction) +[TIP,caption=Release highlights] +==== +* Highlight 1 +* Highlight 2 +* Highlight 3 +==== + ==== New platform features +All subsections must follow this order: + +* Apache Airflow +* Apache Druid +* Apache HBase +* Apache Hadoop +* Apache Hive +* Apache Kafka +* Apache NiFi +* Apache Omid +* Apache Phoenix +* Apache Spark +* Apache Superset +* Apache ZooKeeper +* OpenSearch +* Open Policy Agent +* Trino +* Vector + +Potential sub headers... + +===== General + +====== Security/Observability/Authorization/Miscellaneous + +===== Apache Project and Stackable -operator + ==== Platform improvements +Mark any breaking changes with *Breaking:*. + +===== General + +====== Vulnerabilities + +YYY CVEs were fixed in the Stackable product images. +This includes X critical and XX high-severity CVEs. + +====== Authorization/Miscellaneous + +===== Apache Project and Stackable -operator + +==== Platform fixes + +===== General + +====== Security/Observability/Authorization/Miscellaneous + +===== Apache Project and Stackable -operator + ==== Platform deprecations -==== Product versions +===== Apache Project and Stackable -operator -As with previous SDP releases, many product images have been updated to their latest versions. -The LTS version has in many cases also been adjusted in line with our xref:ROOT:policies.adoc[support policy]. +==== Platform removals + +Mark any breaking changes with *Breaking:*. + +===== General +===== Apache Project and Stackable -operator + +==== Supported versions + +===== Product versions + +As with previous SDP releases, many product images have been updated to their latest versions. Refer to the xref:operators:supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. -===== New versions +====== New LTS versions + +The following product versions were already available before but are now marked as the xref:compliance:policies.adoc#_product_lifecycle_policy[LTS] version: + +* Product 1: https://example.org[1.2.3 (LTS)] +* Product 2: https://example.org[1.2.3 (LTS)] + +====== New versions The following new product versions are now supported: -* ... +* Product 1: https://example.org[1.2.3] +* Product 2: https://example.org[1.2.3 (LTS)] +* Product 2: https://example.org[1.2.3 (experimental)] -===== Deprecated versions +====== Deprecated versions The following product versions are deprecated and will be removed in a later release: -* ... - -===== Removed versions +* Product 1: https://example.org[1.2.3] +* Product 2: https://example.org[1.2.3] +* Product 2: https://example.org[1.2.3] -The following product versions are no longer supported (although images for released product versions remain available https://oci.stackable.tech/[here{external-link-icon}^]. Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here,window=_blank].): +====== Removed versions -* ... +The following product versions are no longer supported. +These images for released product versions remain available https://oci.stackable.tech/[here,window=_blank]. +Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[here,window=_blank]. -==== stackablectl +* Product 1: https://example.org[1.2.3] +* Product 2: https://example.org[1.2.3] +* Product 2: https://example.org[1.2.3] -==== Supported Kubernetes versions +===== Kubernetes versions This release supports the following Kubernetes versions: -* `1.XX` +* `1.33` +* `1.32` +* `1.31` +* `1.30` These Kubernetes versions are no longer supported: -* `1.XX` +* `1.29` -==== Supported OpenShift versions +===== OpenShift versions This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions: -* `4.XX` +* `4.18` +* `4.17` +* `4.16` These OpenShift versions are no longer supported: -* `4.XX` - -==== Breaking changes - -Of the changes mentioned above, the following are breaking (or could lead to breaking behaviour), and you will need to adapt your existing CRDs accordingly: - -===== Stackable Operator for Example Product - -* Description of the change 1 -* Description of the change 2 - -.Breaking changes details -[%collapsible] -==== -* `spec.a`: This field has been removed. -* `spec.b`: This field has been changed to a number. -==== +* `4.15` +* `4.14` ==== Upgrade from OO.M @@ -108,7 +176,7 @@ Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) install The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using `kubectl replace`. -[source] +[source,shell] ---- kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/YY.M.X/deploy/helm/airflow-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/YY.M.X/deploy/helm/commons-operator/crds/crds.yaml @@ -120,14 +188,13 @@ kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operato kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/YY.M.X/deploy/helm/listener-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/YY.M.X/deploy/helm/nifi-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/YY.M.X/deploy/helm/opa-operator/crds/crds.yaml -kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/YY.M.X/deploy/helm/secret-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/YY.M.X/deploy/helm/spark-k8s-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/YY.M.X/deploy/helm/superset-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/YY.M.X/deploy/helm/trino-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/YY.M.X/deploy/helm/zookeeper-operator/crds/crds.yaml ---- -[source,console] +[source] ---- customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced @@ -164,7 +231,7 @@ Afterward you will need to upgrade the CustomResourceDefinitions (CRDs) installe The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using `kubectl replace`. -[source] +[source,shell] ---- kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/YY.M.X/deploy/helm/airflow-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/YY.M.X/deploy/helm/commons-operator/crds/crds.yaml @@ -176,14 +243,13 @@ kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operato kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/YY.M.X/deploy/helm/listener-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/YY.M.X/deploy/helm/nifi-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/YY.M.X/deploy/helm/opa-operator/crds/crds.yaml -kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/YY.M.X/deploy/helm/secret-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/YY.M.X/deploy/helm/spark-k8s-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/YY.M.X/deploy/helm/superset-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/YY.M.X/deploy/helm/trino-operator/crds/crds.yaml kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/YY.M.X/deploy/helm/zookeeper-operator/crds/crds.yaml ---- -[source,console] +[source] ---- customresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced @@ -193,7 +259,8 @@ customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" Install the `YY.M` release -NOTE: `helm repo` subcommands are not supported for OCI registries. The operators are installed directly, without adding the Helm Chart repository first. +NOTE: `helm repo` subcommands are not supported for OCI registries. +The operators are installed directly, without adding the Helm Chart repository first. [source,console] ---- @@ -214,4 +281,7 @@ helm install --wait trino-operator oci://oci.stackable.tech/sdp-charts/trino-ope helm install --wait zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version YY.M.X ---- +[#known-issues-YY_M_X] ==== Known issues + +Note down any potential known issues. From 5c637078714765eab55c341d664b4342592bbeae Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 15:47:31 +0100 Subject: [PATCH 39/54] docs: Update product order in release notes template --- modules/ROOT/partials/release-notes/release-template.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-template.adoc b/modules/ROOT/partials/release-notes/release-template.adoc index 6166d3d9f..459784ed6 100644 --- a/modules/ROOT/partials/release-notes/release-template.adoc +++ b/modules/ROOT/partials/release-notes/release-template.adoc @@ -31,8 +31,8 @@ All subsections must follow this order: * Apache Spark * Apache Superset * Apache ZooKeeper -* OpenSearch * Open Policy Agent +* OpenSearch * Trino * Vector From 86616b20985ceba7f776ce1308aa40df07c7a61a Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 15:50:09 +0100 Subject: [PATCH 40/54] docs: Add Stackable operators to release notes template --- modules/ROOT/partials/release-notes/release-template.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-template.adoc b/modules/ROOT/partials/release-notes/release-template.adoc index 459784ed6..f852d7870 100644 --- a/modules/ROOT/partials/release-notes/release-template.adoc +++ b/modules/ROOT/partials/release-notes/release-template.adoc @@ -35,6 +35,9 @@ All subsections must follow this order: * OpenSearch * Trino * Vector +* Stackable commons-operator +* Stackable listener-operator +* Stackable secret-operator Potential sub headers... From 473fb7aaecb14312bea4300e947df889dadde7c7 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 15:53:33 +0100 Subject: [PATCH 41/54] docs: Adjust product ordering --- modules/ROOT/partials/release-notes/release-25.11.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 27c42d866..5be5ce196 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -586,8 +586,8 @@ The following new product versions are now supported: * Apache Spark: https://github.com/stackabletech/docker-images/issues/1273[3.5.7 (LTS)], https://github.com/stackabletech/docker-images/issues/1273[4.0.1 (experimental)] * Apache Superset: https://github.com/stackabletech/docker-images/issues/1274[4.1.4 (LTS)] * Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1275[3.9.4 (LTS)] -* OpenSearch: https://github.com/stackabletech/docker-images/issues/1239[3.1.0 (LTS)] * Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1238[1.8.0] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1239[3.1.0 (LTS)] * Trino: https://github.com/stackabletech/docker-images/issues/1242[477 (LTS)] * Vector: https://github.com/stackabletech/docker-images/issues/1233[0.49.0] From 33902c55bcb6e0ff2e9c40bff0195e3cbdd5925b Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 16:49:18 +0100 Subject: [PATCH 42/54] docs: Add more small tweaks --- modules/ROOT/partials/release-notes/release-25.11.adoc | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 5be5ce196..569ce89bc 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -61,6 +61,7 @@ See https://github.com/stackabletech/demos/pull/289[demos#289]. This release adds experimental support for KRaft managed Kafka clusters. KRaft Controllers can be deployed instead of Apache ZooKeeper to manage the state of Kafka. +// TODO: Should this instead be "... the only cluster management option available."? KRaft is supported by all Kafka versions provided by SDP, and starting with Kafka 4.X it is the only cluster management version available. See https://github.com/stackabletech/kafka-operator/pull/889[kafka-operator#889]. @@ -88,7 +89,7 @@ Additionally, the operator exposes more Prometheus metrics, such as successful o ===== OpenSearch -The Stackable platform now provides an operator for OpenSearch. +The Stackable Data Platform now provides an operator for OpenSearch. We initially support version link:#new-product-versions-25_11_0[3.1.0], which is also marked as the LTS line going forward. OpenSearch is a powerful search and analytics engine built on Apache Lucene. @@ -234,7 +235,7 @@ See https://github.com/stackabletech/nifi-operator/issues/813[nifi-operator#813] Reduce severity of Pod eviction errors. Previously, the operator would produce lot's of `Cannot evict pod as it would violate the pod's disruption budget` errors. -With this fix, the error is reduced to an info instead. +With this change, the error is reduced to an info instead. See https://github.com/stackabletech/commons-operator/pull/372[commons-operator#372]. ===== Stackable listener-operator @@ -244,7 +245,7 @@ See https://github.com/stackabletech/commons-operator/pull/372[commons-operator# * *Breaking:* The listener-operator default preset changed from `stable-nodes` to `ephemeral-nodes`. Previously, `external-stable` NodePorts pinned the Pod to a specific node, which caused problems with node rotations. // TODO: See if we use code block here or not - Node pinning can be enabled via ListenerClass' new xref:listener-operator:listenerclass.adoc##_node_stickiness[`.spec.pinnedNodePorts`] field. + Node pinning can be enabled via ListenerClass' new xref:listener-operator:listenerclass.adoc#_node_stickiness[`.spec.pinnedNodePorts`] field. The preset can be configured using the following two commands: + -- From f8a405c8d2a6b03971367f41132624cfada525b3 Mon Sep 17 00:00:00 2001 From: Techassi Date: Wed, 12 Nov 2025 16:49:39 +0100 Subject: [PATCH 43/54] docs: Mention how CRD maintenance can be disabled --- .../partials/release-notes/release-25.11.adoc | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 569ce89bc..ac13628ab 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -508,6 +508,30 @@ See the xref:secret-operator:secretclass.adoc#ca-rotation[SecretClass] and xref: The resources are automatically converted by the operator. ** The operator now deploys the CRDs for SecretClass and TrustStore by itself instead of relying on the Helm chart. This enables the operator to automatically rotate and update the TLS certificate (`caBundle`) used for the conversion webhook. + The maintenance of CRDs (and default custom resources) can be disabled via Helm: ++ +-- +[source,yaml] +---- +maintenance: + customResourceDefinitions: + maintain: false +---- + +[WARNING] +==== +When CRD maintenance is disabled, the operator will *not* deploy and manage the CRD. +The CRDs need to be deployed manually and the conversion webhook is disabled. +As a result, only `v1alpha1` SecretClasses can be used. +Only use this setting if you know what you are doing! +==== + +[NOTE] +==== +Currently the maintenance of CRDs and the deployment of default custom resources, such as the `tls` SecretClass are tied together. +This is slated to be changed in an upcoming SDP release. +==== +-- + See https://github.com/stackabletech/secret-operator/pull/634[secret-operator#634]. From b533a2b5f571b55cc5627adffc0c1cb2bb087516 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 14:26:27 +0100 Subject: [PATCH 44/54] chore: Apply suggestions Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com> --- .../partials/release-notes/release-25.11.adoc | 38 +++++++++---------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index ac13628ab..45ece9098 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -62,7 +62,7 @@ See https://github.com/stackabletech/demos/pull/289[demos#289]. This release adds experimental support for KRaft managed Kafka clusters. KRaft Controllers can be deployed instead of Apache ZooKeeper to manage the state of Kafka. // TODO: Should this instead be "... the only cluster management option available."? -KRaft is supported by all Kafka versions provided by SDP, and starting with Kafka 4.X it is the only cluster management version available. +KRaft is supported by all Kafka versions provided by SDP, and starting with Kafka 4 it is the only cluster management version available. See https://github.com/stackabletech/kafka-operator/pull/889[kafka-operator#889]. ===== Apache NiFi @@ -75,11 +75,11 @@ See https://github.com/stackabletech/nifi-operator/issues/812[nifi-operator#812] ===== Apache Spark * The service account of spark applications can now be overridden with pod overrides. - Previously, the application service account was passed as command line argument to spark-submit and it was therefor not possible to overwrite it with pod overrides for the driver and executors. + Previously, the application service account was passed as command line argument to spark-submit and it was therefore not possible to overwrite it with pod overrides for the driver and executors. This CLI argument has now been moved to the pod templates of the individual roles. See https://github.com/stackabletech/spark-k8s-operator/pull/617[spark-k8s-operator#617]. -* This release adds experimental support for Spark 4.X.X. - The support is marked as experimental because Spark 4.0.0 has known compatibility issues with https://github.com/apache/hbase-connectors/pull/130[Apache HBase] and https://github.com/apache/iceberg/issues/13358[Apache Iceberg]. +* This release adds experimental support for Spark 4.0.1. + The support is marked as experimental because Spark 4.0.1 has known compatibility issues with https://github.com/apache/hbase-connectors/pull/130[Apache HBase] and https://github.com/apache/iceberg/issues/13358[Apache Iceberg]. See https://github.com/stackabletech/spark-k8s-operator/issues/586[spark-k8s-operator#586]. ===== Open Policy Agent @@ -150,13 +150,13 @@ In addition to the annotation changes listed above, the following changes were m -- [WARNING] ==== -Care needs to be taken because the metrics format changed. +Care needs to be taken because the metrics format has changed. ==== See https://github.com/stackabletech/hdfs-operator/pull/721[hdfs-operator#721]. -- -* Apache Kafka: Additionally, the `--` Service was replaced with `---headless` and `---metrics` Services. +* Apache Kafka: The `--` Service was replaced with `---headless` and `---metrics` Services. See https://github.com/stackabletech/kafka-operator/pull/897[kafka-operator#897]. ====== Miscellaneous @@ -188,13 +188,13 @@ csiNodeDriver: priorityClassName: ... ---- -* Previously, log entries for some supported products were broken from time to time. +* Previously, log entries for some supported products were occasionally corrupted. These issues have now been resolved by implementing multiple fixes in various affected (upstream) projects. See the tracking issue https://github.com/stackabletech/issues/issues/778[issues#778] for more details. ** Pull request https://github.com/vectordotdev/vector/pull/24028[vectordotdev/vector#24028] was raised to fix log entries with multi-char delimiters. At the time of writing, this PR has not been merged yet, but the fix is manually applied as a patch. See https://github.com/stackabletech/docker-images/pull/1323[docker-images#1323]. -** A XMLLayout multithreading issue in logback has been fixed by raising https://github.com/qos-ch/logback/pull/978[qos-ch/logback#978]. +** An XMLLayout multithreading issue in logback has been fixed by raising https://github.com/qos-ch/logback/pull/978[qos-ch/logback#978]. This fix has been rolled out in all affected products: *** Apache Kafka: https://github.com/stackabletech/docker-images/pull/1330[docker-images#1330] *** Apache NiFi: https://github.com/stackabletech/docker-images/pull/1314[docker-images#1314] @@ -203,7 +203,7 @@ priorityClassName: ... ===== Apache Airflow // TODO: Is there only ONE key or should we clarify WHICH key this is? -* The JWT key is now created by the the operator internally. +* The JWT key is now created internally by the the operator. The same applies to the key previously defined in the credentials secret under `connections.secretKey`: this change is non-breaking, as `connections.secretKey` will be ignored if supplied. See https://github.com/stackabletech/airflow-operator/pull/686[airflow-operator#686]. * Database initialization routines - which are idempotent and run by default - can be deactivated to e.g. help diagnose or troubleshoot start-up issues via the new `databaseInitialization.enabled` field. @@ -213,15 +213,15 @@ priorityClassName: ... Turning off these routines is an unsupported operation as subsequent updates to a running Airflow cluster can result in broken behaviour due to inconsistent metadata. Only use this setting if you know what you are doing! ==== -* The Airflow xref:airflow:index.adoc#_dag_processors[DAG-processor] component now has an optional individual role in the resource definition, allowing it to be separately configured (e.g. logging, resources) and run in a dedicated container. +* The Airflow xref:airflow:index.adoc#_dag_processors[DAG-processor] component now has an optional individual role in the CRD, allowing it to be separately configured (e.g. logging, resources) and run in a dedicated container. See https://github.com/stackabletech/airflow-operator/issues/637[airflow-operator#637]. -* Previously in set-ups where multiple Web-/API-servers were used, only one instance was able to automatically access the connection passwords stored in the database. +* Previously in setups where multiple Web/API-servers were used, only one instance was able to automatically access the connection passwords stored in the database. This could be solved by setting the https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#fernet-key[fernet] key explicitly, but now this detail is taken care of internally by the operator. See https://github.com/stackabletech/airflow-operator/issues/694[airflow-operator#694]. ===== Apache NiFi -The Apache NiFi xref:nifi:usage_guide/monitoring.adoc#_configure_metrics_in_nifi_2_x_x[monitoring documentation] page has been updated to include guidance on how to scrape NiFi 2.X.X metrics using mTLS. +The Apache NiFi xref:nifi:usage_guide/monitoring.adoc#_configure_metrics_in_nifi_2_x_x[monitoring documentation] page has been updated to include guidance on how to scrape NiFi 2 metrics using mTLS. See https://github.com/stackabletech/nifi-operator/issues/813[nifi-operator#813]. ===== Open Policy Agent @@ -233,16 +233,16 @@ See https://github.com/stackabletech/nifi-operator/issues/813[nifi-operator#813] ===== Stackable commons-operator -Reduce severity of Pod eviction errors. -Previously, the operator would produce lot's of `Cannot evict pod as it would violate the pod's disruption budget` errors. -With this change, the error is reduced to an info instead. +Reduce severity of Pod eviction error logs. +Previously, the operator would produce a lot of `ERROR` level logs containing `Cannot evict pod as it would violate the pod's disruption budget`. +With this change, the log level is reduced to `INFO`. See https://github.com/stackabletech/commons-operator/pull/372[commons-operator#372]. ===== Stackable listener-operator * *Breaking:* Default ListenerClass `.spec.externalTrafficPolicy` to `null` to improve LoadBalancer support across various Kubernetes environments. See https://github.com/stackabletech/listener-operator/pull/347[listener-operator#347]. -* *Breaking:* The listener-operator default preset changed from `stable-nodes` to `ephemeral-nodes`. +* *Breaking:* The listener-operator Helm chart default values for `preset` changed from `stable-nodes` to `ephemeral-nodes`. Previously, `external-stable` NodePorts pinned the Pod to a specific node, which caused problems with node rotations. // TODO: See if we use code block here or not Node pinning can be enabled via ListenerClass' new xref:listener-operator:listenerclass.adoc#_node_stickiness[`.spec.pinnedNodePorts`] field. @@ -264,7 +264,7 @@ It should be noted that `stackablectl` automatically detects k3s and kind cluste See the tracking issue https://github.com/stackabletech/issues/issues/770[issues#770] for more details. -- -* *Breaking:* Helm values have changed to allow for separate configuration of affinity, resource, etc... between the operator deployment and the CSI driver pods. +* *Breaking:* Helm values have changed to allow for separate configuration of affinity, resource, etc... between the CSI Provisioner Deployment Pods and the CSI driver DaemonSet Pods. + -- // TODO: Confirm these are the final values for 25.11.0 @@ -388,7 +388,7 @@ See https://github.com/stackabletech/listener-operator/pull/334[listener-operato * *Breaking:* The Helm Chart now deploys the secret-operator as two parts. This separation is needed for CRD versioning and conversion by the operator. ** The controller (which reconciles resources, maintains CRDs and provides the CRD conversion webhook) runs as a Deployment with a single replica. -** The CSI server runs on every Kubernetes cluster node via a DaemonSet (this behaviour is unchanged). +** The CSI Provisioner and Driver runs on every Kubernetes cluster node via a DaemonSet (this behaviour is unchanged). ** The old DaemonSet `secret-operator-daemonset` should be removed manually. ** The Helm values are adjusted in accordance to the changes above. + @@ -520,7 +520,7 @@ maintenance: [WARNING] ==== -When CRD maintenance is disabled, the operator will *not* deploy and manage the CRD. +When CRD maintenance is disabled, the operator will *not* deploy and manage the CRDs. The CRDs need to be deployed manually and the conversion webhook is disabled. As a result, only `v1alpha1` SecretClasses can be used. Only use this setting if you know what you are doing! From 0daf2a4b643bc61688a7bb3474583d362607edb2 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 14:37:54 +0100 Subject: [PATCH 45/54] docs: Adjust casing for some terms --- modules/ROOT/partials/release-notes/release-25.11.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 45ece9098..e6c3581bd 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -74,9 +74,9 @@ See https://github.com/stackabletech/nifi-operator/issues/812[nifi-operator#812] ===== Apache Spark -* The service account of spark applications can now be overridden with pod overrides. - Previously, the application service account was passed as command line argument to spark-submit and it was therefore not possible to overwrite it with pod overrides for the driver and executors. - This CLI argument has now been moved to the pod templates of the individual roles. +* The ServiceAccount of spark applications can now be overridden with `podOverrides`. + Previously, the application ServiceAccount was passed as command line argument to spark-submit and it was therefore not possible to overwrite it with `podOverrides` for the driver and executors. + This CLI argument has now been moved to the Pod templates of the individual roles. See https://github.com/stackabletech/spark-k8s-operator/pull/617[spark-k8s-operator#617]. * This release adds experimental support for Spark 4.0.1. The support is marked as experimental because Spark 4.0.1 has known compatibility issues with https://github.com/apache/hbase-connectors/pull/130[Apache HBase] and https://github.com/apache/iceberg/issues/13358[Apache Iceberg]. From 4eb9c58f98af6e712fdd057d442d65a5321fb571 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 14:38:29 +0100 Subject: [PATCH 46/54] docs: Mention argo-cd-git-ops demo --- modules/ROOT/partials/release-notes/release-25.11.adoc | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index e6c3581bd..ae4b7ebc3 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -45,9 +45,12 @@ See https://github.com/stackabletech/issues/issues/733[issues#733]. ====== Miscellaneous // TODO: Do we want to include this? -The performance of the Trino rules in the `end-to-end-security` stack was improved. +* The performance of the Trino rules in the `end-to-end-security` stack was improved. Batch queries are now significantly faster. See https://github.com/stackabletech/demos/pull/289[demos#289]. +* A new demo has been added, showcasing the interaction between the Stackable Data Platform and ArgoCD to deploy resources managed in Git. + The xref:demos:argo-cd-git-ops.adoc[argo-cd-git-ops] demo deploys Stackable operators and Airflow via ArgoCD, uses Sealed Secrets to safely deploy secrets and credentials and synchronizes Airflow DAGs via Git. + See https://github.com/stackabletech/demos/pull/205[demos#205]. ===== Apache Airflow From 6999bc7ce2819baf808292db9ff58b2f86cebef2 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 14:38:56 +0100 Subject: [PATCH 47/54] docs: Add ... to YAML code blocks --- .../partials/release-notes/release-25.11.adoc | 114 +++++++++--------- 1 file changed, 57 insertions(+), 57 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index ae4b7ebc3..14a370ec6 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -277,12 +277,12 @@ Container resources for the CSI Controller Service (`sdp/listener-operator` in t ---- # Before controller: - resources: + resources: ... # After csiProvisioner: controllerService: - resources: + resources: ... ---- Container image/resources for the external-provisioner (`sig-storage/csi-provisioner` in the Deployment): @@ -291,14 +291,14 @@ Container image/resources for the external-provisioner (`sig-storage/csi-provisi ---- # Before csiProvisioner: - image: - resources: + image: ... + resources: ... # After csiProvisioner: externalProvisioner: - image: - resources: + image: ... + resources: ... ---- Container resources for the CSI Node Service (`sdp/listener-operator` in the DaemonSet): @@ -308,12 +308,12 @@ Container resources for the CSI Node Service (`sdp/listener-operator` in the Dae # Before node: driver: - resources: + resources: ... # After csiNodeDriver: nodeService: - resources: + resources: ... ---- Container image/resources for the node-driver-registrar (`sig-storage/csi-node-driver-registrar` in the DaemonSet): @@ -322,14 +322,14 @@ Container image/resources for the node-driver-registrar (`sig-storage/csi-node-d ---- # Before csiNodeDriverRegistrar: - image: - resources: + image: ... + resources: ... # After csiNodeDriver: nodeDriverRegistrar: - image: - resources: + image: ... + resources: ... ---- Settings that are now split: @@ -337,33 +337,33 @@ Settings that are now split: [source,yaml] ---- # Before -podAnnotations: -podSecurityContext: -securityContext: -nodeSelector: -tolerations: -affinity: +podAnnotations: ... +podSecurityContext: ... +securityContext: ... +nodeSelector: ... +tolerations: ... +affinity: ... # After csiProvisioner: - podAnnotations: - podSecurityContext: - nodeSelector: - tolerations: - affinity: + podAnnotations: ... + podSecurityContext: ... + nodeSelector: ... + tolerations: ... + affinity: ... controllerService: - securityContext: + securityContext: ... csiNodeDriver: - podAnnotations: - podSecurityContext: - nodeSelector: - tolerations: - affinity: + podAnnotations: ... + podSecurityContext: ... + nodeSelector: ... + tolerations: ... + affinity: ... nodeService: - securityContext: + securityContext: ... ---- See the tracking issue https://github.com/stackabletech/issues/issues/763[issues#763] and https://github.com/stackabletech/listener-operator/pull/334[listener-operator#334] for more details. @@ -402,17 +402,17 @@ Both the external provisioner and the node driver registrar have been moved unde ---- # Before csiProvisioner: - resources: + resources: ... csiNodeDriverRegistrar: - resources: + resources: ... # After csiNodeDriver: externalProvisioner: - resources: + resources: ... nodeDriverRegistrar: - resources: + resources: ... ---- The secret-operator is now deployed through a Deployment and a DaemonSet. @@ -423,15 +423,15 @@ As such, the resources of both secret-operator instances can be controlled separ # Before node: driver: - resources: + resources: ... # After csiNodeDriver: nodeService: - resources: + resources: ... controllerService: - resources: + resources: ... ---- The `securityContext` has been split into two parts: @@ -439,15 +439,15 @@ The `securityContext` has been split into two parts: [source,yaml] ---- # Before -securityContext: +securityContext: ... # After csiNodeDriver: nodeService: - securityContext: + securityContext: ... controllerService: - securityContext: + securityContext: ... ---- Settings that are now split: @@ -455,26 +455,26 @@ Settings that are now split: [source,yaml] ---- # Before -podAnnotations: -podSecurityContext: -nodeSelector: -tolerations: -affinity: +podAnnotations: ... +podSecurityContext: ... +nodeSelector: ... +tolerations: ... +affinity: ... # After csiNodeDriver: - podAnnotations: - podSecurityContext: - nodeSelector: - tolerations: - affinity: + podAnnotations: ... + podSecurityContext: ... + nodeSelector: ... + tolerations: ... + affinity: ... controllerService: - podAnnotations: - podSecurityContext: - nodeSelector: - tolerations: - affinity: + podAnnotations: ... + podSecurityContext: ... + nodeSelector: ... + tolerations: ... + affinity: ... ---- Settings that have moved: @@ -482,11 +482,11 @@ Settings that have moved: [source,yaml] ---- # Before -kubeletDir: +kubeletDir: ... # After csiNodeDriver: - kubeletDir: + kubeletDir: ... ---- -- From 72dbc1418f51c38294935f09291121827ec611c2 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 14:50:38 +0100 Subject: [PATCH 48/54] docs: Mention that secret-operator CRDs don't need be replaced manually --- .../ROOT/partials/release-notes/release-25.11.adoc | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 14a370ec6..bcb338db3 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -386,6 +386,7 @@ It should be noted that generally no action is required, but that depends on whe + See https://github.com/stackabletech/listener-operator/pull/334[listener-operator#334] for more details. +[#secret-operator-improvements-25_11_0] ===== Stackable secret-operator * *Breaking:* The Helm Chart now deploys the secret-operator as two parts. @@ -709,7 +710,10 @@ Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) install The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using `kubectl replace`. -// TODO: Mention that the CRDs of the secret-operator don't need to be replaced manually. +[NOTE] +==== +It should be noted that the SecretClass and TrustStore CRDs don't need to be replaced manually, because the Stackable secret-operator link:#secret-operator-improvements-25_11_0[maintains them by default]. +==== [source,shell] ---- @@ -766,7 +770,10 @@ Afterward you will need to upgrade the CustomResourceDefinitions (CRDs) installe The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using `kubectl replace`. -// TODO: Mention that the CRDs of the secret-operator don't need to be replaced manually. +[NOTE] +==== +It should be noted that the SecretClass and TrustStore CRDs don't need to be replaced manually, because the Stackable secret-operator link:#secret-operator-improvements-25_11_0[maintains them by default]. +==== [source,shell] ---- From 24af871a46b9ca902c63ff64b866d47756c3243d Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 15:40:43 +0100 Subject: [PATCH 49/54] docs: Mention secret-operator DaemonSet renames --- .../ROOT/partials/release-notes/release-25.11.adoc | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index bcb338db3..0239153e4 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -393,7 +393,6 @@ See https://github.com/stackabletech/listener-operator/pull/334[listener-operato This separation is needed for CRD versioning and conversion by the operator. ** The controller (which reconciles resources, maintains CRDs and provides the CRD conversion webhook) runs as a Deployment with a single replica. ** The CSI Provisioner and Driver runs on every Kubernetes cluster node via a DaemonSet (this behaviour is unchanged). -** The old DaemonSet `secret-operator-daemonset` should be removed manually. ** The Helm values are adjusted in accordance to the changes above. + -- @@ -492,6 +491,16 @@ csiNodeDriver: -- +** As part of the Helm value changes listed above, some resource names have also been updated. ++ +[WARNING] +==== +It should be noted that generally no action is required, but that depends on whether or not your deployment scripts (eg: Kustomize) or monitoring/alerting system depends on any of the names and values. +==== +*** DaemonSet `secret-operator-daemonset` has been renamed to `secret-operator-csi-node-driver` +**** Container `secret-operator` has been renamed to `csi-node-service` + + + See https://github.com/stackabletech/secret-operator/pull/645[secret-operator#645]. From 3e9abcb526c04d559b44a44ff3a54f9f0b98506c Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 15:44:50 +0100 Subject: [PATCH 50/54] chore: Apply suggestion Co-authored-by: Andrew Kenworthy <1712947+adwk67@users.noreply.github.com> --- modules/ROOT/partials/release-notes/release-25.11.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index 0239153e4..f2be466f5 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -635,7 +635,7 @@ The following product versions are deprecated and will be removed in a later rel * Apache Airflow: https://github.com/stackabletech/docker-images/issues/1266[2.9.3], https://github.com/stackabletech/docker-images/issues/1268[2.10.5] * Apache Druid: https://github.com/stackabletech/docker-images/issues/1234[33.0.0] -* Apache HBase: https://github.com/stackabletech/docker-images/issues/1269[2.6.4] +* Apache HBase: https://github.com/stackabletech/docker-images/issues/1269[2.6.2] * Apache Hadoop: https://github.com/stackabletech/docker-images/issues/1270[3.4.1] * Apache Hive: https://github.com/stackabletech/docker-images/issues/1271[4.0.0] * Apache Kafka: https://github.com/stackabletech/docker-images/issues/1236[3.7.2] From 0c2d4530dc1c7d54fb494cac583c6fe916c902b7 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 16:05:41 +0100 Subject: [PATCH 51/54] docs: Update getting-started page --- modules/ROOT/pages/getting-started.adoc | 28 ++++++++++++------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/modules/ROOT/pages/getting-started.adoc b/modules/ROOT/pages/getting-started.adoc index a19ce1a66..d14c340b9 100644 --- a/modules/ROOT/pages/getting-started.adoc +++ b/modules/ROOT/pages/getting-started.adoc @@ -37,11 +37,11 @@ These can be installed on any node that has access to the Kubernetes control pla In this example we will install them on the controller node. Stackable operators can be installed using `stackablectl`. -Run the following commands to install ZooKeeper, Kafka and NiFi from the Stackable 25.7 release. +Run the following commands to install ZooKeeper, Kafka and NiFi from the Stackable 25.11 release. [source,bash] ---- -stackablectl release install -i commons -i secret -i listener -i zookeeper -i kafka -i nifi 25.7 +stackablectl release install -i commons -i secret -i listener -i zookeeper -i kafka -i nifi 25.11 ---- .Using Helm instead @@ -56,12 +56,12 @@ Install the operators: [source,bash] ---- -helm install zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version=25.7.0 -helm install kafka-operator oci://oci.stackable.tech/sdp-charts/kafka-operator --version=25.7.0 -helm install secret-operator oci://oci.stackable.tech/sdp-charts/secret-operator --version=25.7.0 -helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --version=25.7.0 -helm install commons-operator oci://oci.stackable.tech/sdp-charts/commons-operator --version=25.7.0 -helm install nifi-operator oci://oci.stackable.tech/sdp-charts/nifi-operator --version=25.7.0 +helm install zookeeper-operator oci://oci.stackable.tech/sdp-charts/zookeeper-operator --version=25.11.0 +helm install kafka-operator oci://oci.stackable.tech/sdp-charts/kafka-operator --version=25.11.0 +helm install secret-operator oci://oci.stackable.tech/sdp-charts/secret-operator --version=25.11.0 +helm install listener-operator oci://oci.stackable.tech/sdp-charts/listener-operator --version=25.11.0 +helm install commons-operator oci://oci.stackable.tech/sdp-charts/commons-operator --version=25.11.0 +helm install nifi-operator oci://oci.stackable.tech/sdp-charts/nifi-operator --version=25.11.0 ---- ==== @@ -70,12 +70,12 @@ You can check which operators are installed using `stackablectl operator install [source,console] ---- OPERATOR VERSION NAMESPACE STATUS LAST UPDATED -commons 25.7.0 default deployed 2025-07-24 17:58:32.916032854 +0100 CET -kafka 25.7.0 default deployed 2025-07-24 17:58:55.036115353 +0100 CET -listener 25.7.0 default deployed 2025-07-24 17:59:18.136775259 +0100 CET -nifi 25.7.0 default deployed 2025-07-24 17:59:51.927081648 +0100 CET -secret 25.7.0 default deployed 2025-07-24 18:00:05.060241771 +0100 CET -zookeeper 25.7.0 default deployed 2025-07-24 18:00:08.425686918 +0100 CET +commons 25.11.0 default deployed 2025-11-07 17:58:32.916032854 +0100 CET +kafka 25.11.0 default deployed 2025-11-07 17:58:55.036115353 +0100 CET +listener 25.11.0 default deployed 2025-11-07 17:59:18.136775259 +0100 CET +nifi 25.11.0 default deployed 2025-11-07 17:59:51.927081648 +0100 CET +secret 25.11.0 default deployed 2025-11-07 18:00:05.060241771 +0100 CET +zookeeper 25.11.0 default deployed 2025-11-07 18:00:08.425686918 +0100 CET ---- == Deploying Stackable Services From b930b2f7f1bc8eda1964e7cc591e527a0bef0e84 Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 16:27:27 +0100 Subject: [PATCH 52/54] docs: Add summary of breaking changes --- .../ROOT/partials/release-notes/release-25.11.adoc | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index f2be466f5..b0a82ce99 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -14,6 +14,15 @@ Released on 2025-11-07. ** The `certManager` backend is no longer marked as experimental. ==== +[WARNING,caption=Summary of breaking changes] +==== +The following components of the SDP contain breaking changes for this release: + +* link:#opa-improvements-25_11_0[Open Policy Agent] +* link:#listener-operator-improvements-25_11_0[Stackable listener-operator] +* link:#secret-operator-improvements-25_11_0[Stackable secret-operator] +==== + ==== New platform features ===== General @@ -227,6 +236,7 @@ Only use this setting if you know what you are doing! The Apache NiFi xref:nifi:usage_guide/monitoring.adoc#_configure_metrics_in_nifi_2_x_x[monitoring documentation] page has been updated to include guidance on how to scrape NiFi 2 metrics using mTLS. See https://github.com/stackabletech/nifi-operator/issues/813[nifi-operator#813]. +[#opa-improvements-25_11_0] ===== Open Policy Agent * *Breaking:* The per-rolegroup Services now only expose the HTTP port and contain a `-headless` suffix to better indicate their purpose and to be consistent with other operators. @@ -241,6 +251,7 @@ Previously, the operator would produce a lot of `ERROR` level logs containing `C With this change, the log level is reduced to `INFO`. See https://github.com/stackabletech/commons-operator/pull/372[commons-operator#372]. +[#listener-operator-improvements-25_11_0] ===== Stackable listener-operator * *Breaking:* Default ListenerClass `.spec.externalTrafficPolicy` to `null` to improve LoadBalancer support across various Kubernetes environments. From fbc6319910818943b9d0e89c5c7ca4646f98fb8f Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 16:27:56 +0100 Subject: [PATCH 53/54] chore: Add a few small tweaks --- modules/ROOT/partials/release-notes/release-25.11.adoc | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index b0a82ce99..f0750217d 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -58,7 +58,7 @@ See https://github.com/stackabletech/issues/issues/733[issues#733]. Batch queries are now significantly faster. See https://github.com/stackabletech/demos/pull/289[demos#289]. * A new demo has been added, showcasing the interaction between the Stackable Data Platform and ArgoCD to deploy resources managed in Git. - The xref:demos:argo-cd-git-ops.adoc[argo-cd-git-ops] demo deploys Stackable operators and Airflow via ArgoCD, uses Sealed Secrets to safely deploy secrets and credentials and synchronizes Airflow DAGs via Git. + The xref:demos:argo-cd-git-ops.adoc[`argo-cd-git-ops`] demo deploys Stackable operators and Airflow via ArgoCD, uses Sealed Secrets to safely deploy secrets and credentials and synchronizes Airflow DAGs via Git. See https://github.com/stackabletech/demos/pull/205[demos#205]. ===== Apache Airflow @@ -214,7 +214,6 @@ priorityClassName: ... ===== Apache Airflow -// TODO: Is there only ONE key or should we clarify WHICH key this is? * The JWT key is now created internally by the the operator. The same applies to the key previously defined in the credentials secret under `connections.secretKey`: this change is non-breaking, as `connections.secretKey` will be ignored if supplied. See https://github.com/stackabletech/airflow-operator/pull/686[airflow-operator#686]. From 991770743e82c42a43ffa5a24b72719f5eb0085f Mon Sep 17 00:00:00 2001 From: Techassi Date: Thu, 13 Nov 2025 16:36:39 +0100 Subject: [PATCH 54/54] chore: Summary -> overview --- modules/ROOT/partials/release-notes/release-25.11.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-25.11.adoc b/modules/ROOT/partials/release-notes/release-25.11.adoc index f0750217d..94e36b564 100644 --- a/modules/ROOT/partials/release-notes/release-25.11.adoc +++ b/modules/ROOT/partials/release-notes/release-25.11.adoc @@ -14,7 +14,7 @@ Released on 2025-11-07. ** The `certManager` backend is no longer marked as experimental. ==== -[WARNING,caption=Summary of breaking changes] +[WARNING,caption=Overview of breaking changes] ==== The following components of the SDP contain breaking changes for this release: