Fix elasticsearch #62

Bueller87 · 2025-11-13T17:00:41Z

Summary

Adds support for deploying Cadence with PostgreSQL + Elasticsearch + Kafka for advanced visibility. This configuration enables powerful workflow search and filtering capabilities through Elasticsearch while using PostgreSQL as the primary persistence layer.

New Example Configuration: `values.postgres-es7.yaml`

This branch introduces a complete, production-ready example for deploying Cadence with:

PostgreSQL - Primary persistence for workflow state and history
Elasticsearch 7 - Advanced visibility with rich search capabilities
Kafka - Asynchronous visibility event processing for better performance
Kafka topic auto-provisioning - Eliminates manual setup and race conditions

Testing

Clean deployment with all pods healthy (no scheduling failures)
Kafka topics auto-provisioned with correct replication factors
Workflows visible in Cadence Web UI immediately after execution
No visibility-related errors in service logs
Repeatable deployments (tested multiple tear-down/redeploy cycles)

Caveats

Single-broker Kafka: Configured for development/testing with replicationFactor: 1. For production, increase Kafka broker count and replication factors to 3+.
Resource requirements: Deploys 9+ pods (Cadence services + PostgreSQL + Elasticsearch + Kafka). Ensure your cluster has adequate resources.
Fresh deployments recommended: If upgrading from a previous configuration, a clean redeploy is recommended to ensure Kafka topics are properly provisioned.

Checklist

[ x] Chart.yaml: Chart version bumped
[ x] values.yaml: global.image.tag (should match with Chart.yaml appVersion)
[ x] Dependencies in Chart.yaml (check for updates with helm dependency update, aim for latest major-1 stable)
[ x] DCO signed
[ x] Test deployment locally
[ x] Run helm-docs to update documentation

) Description of this PR schema upgrade job should fail open on db creation to handle already exist errors add example of cadence with postgres Local Test namespace and release name is still fixed unfortunately. cd charts/cadence helm install cadence-release . -n cadence-postgres --values examples/values.postgres.yaml Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

…dence-workflow#57) * docs: Convert slack links to CNCF, add link to contributing guide Signed-off-by: vpatil16@ext.uber.com <vpatil16@ext.uber.com> Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Signed-off-by: CosminL-DEV <82363564+CosminL-DEV@users.noreply.github.com> Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

When using Elasticsearch for visibility, the PostgreSQL visibility database is not needed. Removed visibilityDbname and initdb script that were creating an unused database. Also updated dynamic config to use es instead of es-visibility to match Cadence's internal visibility store naming convention. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

The schema job should only create the SQL/Cassandra visibility database when Elasticsearch is disabled. When using Elasticsearch for advanced visibility, the SQL visibility database is not needed. Changed condition from checking if visibilityDbname is set to checking if ES_ENABLED is false, allowing ES-only deployments to skip unnecessary database creation. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Changed visibilityIndex from 'cadence-visibility' to 'cadence-visibility-v1' to match the Elasticsearch template pattern 'cadence-visibility-*'. This ensures the template with proper keyword field mappings is applied to the index, fixing visibility queries that require exact term matching on fields like DomainID. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Set visibilityStore to empty string to prevent dual-write mode when using Elasticsearch-only advanced visibility. This fixes 'Operation is not supported' errors when Cadence tries to write to a non-existent SQL visibility database. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

The visibility store name in dynamic config should be 'es' (a special keyword for advanced visibility), not the datastore name 'es-visibility'. This matches the Cadence upstream configuration pattern where advancedVisibilityStore is 'es-visibility' but system.writeVisibilityStoreName is 'es'. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Removed advancedVisibilityStore from persistence config to enable Kafka-based async visibility mode. When advancedVisibilityStore is set, Cadence tries to use dual-write mode (SQL + ES) which fails without a SQL visibility database. With Kafka mode, the history service writes to Kafka, and the worker service reads from Kafka and writes to ES. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Must explicitly override the default value of 'es-visibility' with an empty string to disable direct ES writes and enable Kafka-based async visibility. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Cadence requires at least one of visibilityStore or advancedVisibilityStore to be set. We keep advancedVisibilityStore='es-visibility' to define the datastore, and use dynamic config system.writeVisibilityStoreName='es' to enable Kafka-based async visibility instead of direct ES writes. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Added Kafka provisioning configuration to pre-create required topics: - __consumer_offsets: Internal Kafka topic for consumer offset tracking - cadence-visibility: Main visibility events topic - cadence-visibility-dlq: Dead letter queue for failed visibility events This prevents race conditions where topics get auto-created with incorrect replication factors before the broker config is fully loaded. All topics are created with replication-factor: 1 to match our single-broker setup. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

shijiesheng · 2025-11-20T20:17:42Z

charts/cadence/templates/server-configmap.yaml

      numHistoryShards: {{ .Values.config.persistence.numHistoryShards | default 4 }}
      defaultStore: {{ .Values.config.persistence.defaultStore | default "default" | quote }}
-      {{- if and .Values.config.persistence.visibilityStore (not .Values.config.persistence.elasticsearch.enabled) }}
+      {{- if .Values.config.persistence.visibilityStore }}


we do need this additional check (not .Values.config.persistence.elasticsearch.enabled)

shijiesheng · 2025-11-20T20:20:21Z

charts/cadence/templates/server-configmap.yaml

-        {{- if ne .Values.config.persistence.elasticsearch.enabled true }}
+        {{- if .Values.config.persistence.visibilityStore }}
        # Visibility datastore
-        {{ if .Values.config.persistence.visibilityStore -}}
-        {{ .Values.config.persistence.visibilityStore | default "visibility" }}:
+        {{ .Values.config.persistence.visibilityStore }}:


there is no issue with the existing checkers. basically it says set vibility table in primary storage if advanced storage is not enabled. Default value of "visibility" also makes sense to minimize the visibility

I might be confused as I don't know exactly how the code works. But I found if .Values.config.persistence.visibilityStore is set to anything. Cadence will write to that store and not the advancedVisibilityStore. So I changed the logic a little, but I should test that I didn't break the plain postgres example values.

charts/cadence/examples/values.postgres-es7.yaml

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

shijiesheng and others added 11 commits November 14, 2025 10:52

docs: Convert slack links to CNCF, add link to contributing guide (ca…

502a562

…dence-workflow#57) * docs: Convert slack links to CNCF, add link to contributing guide Signed-off-by: vpatil16@ext.uber.com <vpatil16@ext.uber.com> Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Bump Web v4.0.11 (cadence-workflow#60)

e63fb49

Signed-off-by: CosminL-DEV <82363564+CosminL-DEV@users.noreply.github.com> Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

add es examples and add to chart dependencies

ecc3e4e

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

add additional comment

4d1be74

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

add kafka values to elasticsearch example

45a7f51

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

fix configmap so visibility datastore is created

9122e55

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

change cadence verion to 1.3.7

481cf79

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Set cadence version to v1.3.6 in chart and values

be5335d

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

fix: always create visibility database when configured

d8bb312

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

run helm-docs to update Readme

c85e226

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Bueller87 force-pushed the fix-elasticsearch branch from 9ca4016 to c85e226 Compare November 14, 2025 18:53

Bueller87 added 10 commits November 17, 2025 14:22

fix: explicitly set advancedVisibilityStore to empty string

297dcad

Must explicitly override the default value of 'es-visibility' with an empty string to disable direct ES writes and enable Kafka-based async visibility. Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Ran helm-docs

4ad9d9d

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

shijiesheng reviewed Nov 20, 2025

View reviewed changes

Bueller87 and others added 2 commits November 20, 2025 13:16

Merge branch 'main' into fix-elasticsearch

47b90b7

fix: change visibility index name

15d3d14

Signed-off-by: “Kevin” <kevlar_ksb@yahoo.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix elasticsearch #62

Fix elasticsearch #62

Bueller87 commented Nov 13, 2025 •

edited

Loading

Uh oh!

shijiesheng Nov 20, 2025

Uh oh!

shijiesheng Nov 20, 2025

Uh oh!

Bueller87 Nov 20, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Fix elasticsearch #62

Are you sure you want to change the base?

Fix elasticsearch #62

Conversation

Bueller87 commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New Example Configuration: values.postgres-es7.yaml

Testing

Caveats

Checklist

Uh oh!

shijiesheng Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

shijiesheng Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Bueller87 Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Bueller87 commented Nov 13, 2025 •

edited

Loading

New Example Configuration: `values.postgres-es7.yaml`

Bueller87 Nov 20, 2025 •

edited

Loading