Infinite reconciliation loop for secrets causing high kube-apiserver load

[vlasov-y]

Describe the bug
Operator reconcile secrets without any delay in infinite loop

To Reproduce
I do not know what causes that, but check logs

Expected behavior
No infinite loop.

Environment:

• Kubernetes version: v1.33.4-eks-e386d34
• kubernetes-replicator version: 2.12.0

Operator manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      serviceAccountName: app
      automountServiceAccountToken: true
      enableServiceLinks: false
      securityContext:
        seccompProfile:
          type: RuntimeDefault
        runAsNonRoot: true
        runAsUser: 65534
        runAsGroup: 65534
        fsGroup: 65534
      containers:
        - name: replicator
          image: replicator
          args:
            - --exclude-namespaces=kube-system,kube-node-lease,kube-public
            - --replicate-configmaps=true
            - --replicate-secrets=true
            - --replicate-role-bindings=false
            - --replicate-roles=false
            - --replicate-service-accounts=false
            - --sync-by-content
            - --resync-period=30m
          securityContext:
            privileged: false
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
            readOnlyRootFilesystem: true
          ports:
            - name: health
              containerPort: 9102
          livenessProbe:
            httpGet:
              path: /healthz
              port: health
          readinessProbe:
            httpGet:
              path: /readyz
              port: health
          resources:
            limits:
              cpu: 200m
              memory: 50Mi
            requests:
              cpu: 10m
              memory: 20Mi
      nodeSelector:
        node-role.cluster.local/system: "true"
      tolerations:
        - key: node-role.cluster.local/system
          effect: NoSchedule

Secret example:

apiVersion: v1
kind: Secret
metadata:
  annotations:
    replicator.v1.mittwald.de/replicate-to-matching: |
      kubernetes.io/metadata.name notin (default, kube-node-lease, kube-system, kube-public, flux-system)
  name: ks-general-variables
  namespace: system
type: Opaque
stringData:
  AWS_REGION: eu-central-1
  BACKUP_BUCKET: redacted
  CLUSTER: test
  WAF_GENERAL_ARN: redacted

Logs:

Pay attention to log timestamps, that happens one after another in a loop.

Logs

time="2025-10-08T10:45:37Z" level=debug msg="using flag values main.flags{Kubeconfig:\"\", ResyncPeriodS:\"30m\", ResyncPeriod:1800000000000, StatusAddr:\":9102\", AllowAll:false, LogLevel:\"debug\", LogFormat:\"plain\", ReplicateSecrets:true, ReplicateConfigMaps:true, ReplicateRoles:false, ReplicateRoleBindings:false, ReplicateServiceAccounts:false, SyncByContent:true, ExcludeNamespaces:\"kube-system,kube-node-lease,kube-public\"}"
time="2025-10-08T10:45:37Z" level=info msg="using in-cluster configuration"
time="2025-10-08T10:45:37Z" level=info msg="running Namespace controller" kind=Namespace
time="2025-10-08T10:45:37Z" level=info msg="starting liveness monitor at :9102"
time="2025-10-08T10:45:37Z" level=info msg="running Secret controller" kind=Secret
time="2025-10-08T10:45:37Z" level=info msg="running ConfigMap controller" kind=ConfigMap
time="2025-10-08T10:45:38Z" level=info msg="Checking if miscellaneous/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:38Z" level=debug msg="Secret miscellaneous/ks-general-variables is already up-to-date" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:38Z" level=info msg="Replicated system/ks-general-variables to: miscellaneous" source=system/ks-general-variables
time="2025-10-08T10:45:38Z" level=info msg="Checking if monitoring/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:38Z" level=debug msg="Secret monitoring/ks-general-variables is already up-to-date" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:38Z" level=info msg="Replicated system/ks-general-variables to: monitoring" source=system/ks-general-variables
time="2025-10-08T10:45:38Z" level=info msg="Checking if pri/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:38Z" level=debug msg="Secret pri/ks-general-variables is already up-to-date" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:38Z" level=info msg="Replicated system/ks-general-variables to: pri" source=system/ks-general-variables
time="2025-10-08T10:45:38Z" level=info msg="Checking if system/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:38Z" level=debug msg="Updating existing secret system/ks-general-variables" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:38Z" level=info msg="Checking if test/ks-general-variables exists? false" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:38Z" level=debug msg="Creating a new secret secret test/ks-general-variables" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:38Z" level=error msg="error while replicating by label selector" error="Replicated system/ks-general-variables to 3 out of 5 namespaces: 2 errors occurred:\n\t* Failed to replicate Secret system/ks-general-variables -> system: Failed to update secret system/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret system/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\t* Failed to replicate Secret system/ks-general-variables -> test: Failed to update secret test/ks-general-variables: secrets \"ks-general-variables\" already exists: Failed to update secret test/ks-general-variables: secrets \"ks-general-variables\" already exists\n\n" kind=Secret resource=system/ks-general-variables
time="2025-10-08T10:45:38Z" level=info msg="Secret system/sops-age to be replicated to: [flux-system]" kind=Secret source=system/sops-age
time="2025-10-08T10:45:38Z" level=info msg="Checking if flux-system/sops-age exists? true" kind=Secret source=system/sops-age target=flux-system/sops-age
time="2025-10-08T10:45:38Z" level=debug msg="Secret flux-system/sops-age is already up-to-date" kind=Secret source=system/sops-age target=flux-system/sops-age
time="2025-10-08T10:45:38Z" level=info msg="Replicated system/sops-age to: flux-system" source=system/sops-age
time="2025-10-08T10:45:39Z" level=info msg="Checking if miscellaneous/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret miscellaneous/ks-general-variables" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:39Z" level=info msg="Checking if monitoring/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret monitoring/ks-general-variables" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:39Z" level=info msg="Checking if pri/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret pri/ks-general-variables" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:39Z" level=info msg="Checking if system/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret system/ks-general-variables" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:39Z" level=info msg="Checking if test/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret test/ks-general-variables" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:39Z" level=error msg="error while replicating by label selector" error="Replicated system/ks-general-variables to 0 out of 5 namespaces: 5 errors occurred:\n\t* Failed to replicate Secret system/ks-general-variables -> miscellaneous: Failed to update secret miscellaneous/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret miscellaneous/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\t* Failed to replicate Secret system/ks-general-variables -> monitoring: Failed to update secret monitoring/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret monitoring/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\t* Failed to replicate Secret system/ks-general-variables -> pri: Failed to update secret pri/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret pri/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\t* Failed to replicate Secret system/ks-general-variables -> system: Failed to update secret system/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret system/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\t* Failed to replicate Secret system/ks-general-variables -> test: Failed to update secret test/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret test/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\n" kind=Secret resource=system/ks-general-variables
time="2025-10-08T10:45:39Z" level=info msg="Checking if miscellaneous/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret miscellaneous/ks-general-variables" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:39Z" level=info msg="Checking if monitoring/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret monitoring/ks-general-variables" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:39Z" level=info msg="Checking if pri/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret pri/ks-general-variables" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:39Z" level=info msg="Checking if system/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret system/ks-general-variables" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:39Z" level=info msg="Checking if test/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:39Z" level=debug msg="Updating existing secret test/ks-general-variables" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:39Z" level=error msg="error while replicating by label selector" error="Replicated system/ks-general-variables to 0 out of 5 namespaces: 5 errors occurred:\n\t* Failed to replicate Secret system/ks-general-variables -> miscellaneous: Failed to update secret miscellaneous/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret miscellaneous/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\t* Failed to replicate Secret system/ks-general-variables -> monitoring: Failed to update secret monitoring/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret monitoring/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\t* Failed to replicate Secret system/ks-general-variables -> pri: Failed to update secret pri/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret pri/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\t* Failed to replicate Secret system/ks-general-variables -> system: Failed to update secret system/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret system/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\t* Failed to replicate Secret system/ks-general-variables -> test: Failed to update secret test/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret test/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\n" kind=Secret resource=system/ks-general-variables
time="2025-10-08T10:45:40Z" level=info msg="Checking if miscellaneous/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:40Z" level=debug msg="Updating existing secret miscellaneous/ks-general-variables" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:40Z" level=info msg="Replicated system/ks-general-variables to: miscellaneous" source=system/ks-general-variables
time="2025-10-08T10:45:40Z" level=info msg="Checking if monitoring/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:40Z" level=debug msg="Updating existing secret monitoring/ks-general-variables" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:40Z" level=info msg="Replicated system/ks-general-variables to: monitoring" source=system/ks-general-variables
time="2025-10-08T10:45:40Z" level=info msg="Checking if pri/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:40Z" level=debug msg="Updating existing secret pri/ks-general-variables" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:40Z" level=info msg="Replicated system/ks-general-variables to: pri" source=system/ks-general-variables
time="2025-10-08T10:45:40Z" level=info msg="Checking if system/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:40Z" level=debug msg="Updating existing secret system/ks-general-variables" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:40Z" level=info msg="Replicated system/ks-general-variables to: system" source=system/ks-general-variables
time="2025-10-08T10:45:40Z" level=info msg="Checking if test/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:40Z" level=debug msg="Updating existing secret test/ks-general-variables" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:41Z" level=error msg="error while replicating by label selector" error="Replicated system/ks-general-variables to 4 out of 5 namespaces: 1 error occurred:\n\t* Failed to replicate Secret system/ks-general-variables -> test: Failed to update secret test/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again: Failed to update secret test/ks-general-variables: Operation cannot be fulfilled on secrets \"ks-general-variables\": the object has been modified; please apply your changes to the latest version and try again\n\n" kind=Secret resource=system/ks-general-variables
time="2025-10-08T10:45:41Z" level=info msg="Checking if miscellaneous/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:41Z" level=debug msg="Updating existing secret miscellaneous/ks-general-variables" kind=Secret source=system/ks-general-variables target=miscellaneous/ks-general-variables
time="2025-10-08T10:45:41Z" level=info msg="Replicated system/ks-general-variables to: miscellaneous" source=system/ks-general-variables
time="2025-10-08T10:45:41Z" level=info msg="Checking if monitoring/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:41Z" level=debug msg="Updating existing secret monitoring/ks-general-variables" kind=Secret source=system/ks-general-variables target=monitoring/ks-general-variables
time="2025-10-08T10:45:41Z" level=info msg="Replicated system/ks-general-variables to: monitoring" source=system/ks-general-variables
time="2025-10-08T10:45:41Z" level=info msg="Checking if pri/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:41Z" level=debug msg="Updating existing secret pri/ks-general-variables" kind=Secret source=system/ks-general-variables target=pri/ks-general-variables
time="2025-10-08T10:45:41Z" level=info msg="Replicated system/ks-general-variables to: pri" source=system/ks-general-variables
time="2025-10-08T10:45:41Z" level=info msg="Checking if system/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:41Z" level=debug msg="Updating existing secret system/ks-general-variables" kind=Secret source=system/ks-general-variables target=system/ks-general-variables
time="2025-10-08T10:45:42Z" level=info msg="Replicated system/ks-general-variables to: system" source=system/ks-general-variables
time="2025-10-08T10:45:42Z" level=info msg="Checking if test/ks-general-variables exists? true" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:42Z" level=debug msg="Updating existing secret test/ks-general-variables" kind=Secret source=system/ks-general-variables target=test/ks-general-variables
time="2025-10-08T10:45:42Z" level=info msg="Replicated system/ks-general-variables to: test" source=system/ks-general-variables