Failure to automatically recover from "secret already exists" state

[kavishdahekar]

Describe the bug
If a secret already exists at a target that the replicator was planning to replicate to, the replication task is skipped with error

Replicated test-source/test-secret-1 to 299 out of 300 namespaces: 1 error occurred:
\t* Failed to replicate Secret test-source/test-secret-1 -> test-destination-299: Failed to update secret test-destination-299/test-secret-1: secrets \"test-secret-1\" already exists: Failed to update secret test-destination-299/test-secret-1: secrets \"test-secret-1\" already exists

Deleting the existing secret (test-destination-299/test-secret-1 in above example) however does not trigger the replicator to create a proper copy. In above case, if test-destination-299/test-secret-1 is deleted, replicator never recreates it (unless the source secret is modified or the replicator pods are restarted).

To Reproduce

1. create a secret in ns 'destination`.
2. create the exact same secret with replicator.v1.mittwald.de/replicate-to=destaination annotation
3. replicator fails with error "already exists"
4. delete secret from destination ns.
5. note that replicator does not create the secret in the destination ns.

Expected behavior
Replicator should observe that the conflicting secret no longer exists and should create the secret in the destination ns.

Environment:

• Kubernetes version: v1.30.5
• kubernetes-replicator version: v2.10.2

Additional context
n/a

[kavishdahekar]

Possible mitigation for this bug could be introduction of a new annotation replicator.v1.mittwald.de/force-replicate=true but this would need more though.

[roderik]

For me it never updates existing secrets, not sure why...

[roderik]

I think part of the reason is we deploy with pulumi which replaces the secrets, so the resource version might always be the same?
The check contents flag in the last release doesn't work for me either

[vasili439]

same for me. if some replication finished with error it will never retry until kubernetes replicator will be restarted.