Explore using the AWS Secrets Manager caching feature

[jicowan]

AWS Secrets Manager has a caching library which updates the cache periodically, ensuring your applications use the most up to date secret value without polling. If you run the init container as a sidecar and make use of this library it may eliminate the need for an operator that recycles the pod when a secret is rotated. The downside is that it will require you to run a sidecar (increases the resource requirements) and your application will have to be written to re-read the password from disk when the secret is rotated.

[allamand]

If we rewrite the application, then I would suggest let the app directly read the secret and not use the Sidecar container. What do you think ?

[jicowan]

@allamand When I say "re-write the application" I mean writing the application to re-read the secret from disk when an error occurs using the old secret. I think that's an easier change than adding the AWS SDK to your application. I'm open to being convinced otherwise.

[jicowan]

Our current plan is to use a CRD/operator to patch the pods in a deployment that references a secret from AWS Secrets Manager when the secret is rotated. The CRD will maintain a mapping between deployment:secret and will inform the operator which messages to act upon. AWS Secret Manager events will be put into an SQS message queue which the operator will read from. When the operator receives a message that indicates a password has been rotated, it will loop through the CRDs until it finds a matching deployment for that secret. It will then patch that deployment which will cause the pods to restart and reload the secret.

[anguslees]

Sounds reasonable. Just to put it out there, I think you've described a "push" pod update mechanism. Another approach might be "pull":

The operator could notify the sidecars when the new secret is available. Eg: by just updating the proposed CRD with a timestamp or other generation number, then all sidecars 'watching' that CRD would go and fetch the latest secret value from AWS, and write it to the volume shared with the main container.

Pros: no need to find+restart Deployments. Cons: sidecar needs a k8s watch on a new CRD.

[jicowan]

@anguslees We were purposely trying to avoid polling because a) there could be a long wait between polling intervals causing the app to break and b) excessive polling could get expensive. We could add a listener to the sidecar that fetches the secret again when it receives a message from the controller or we could subscribe it to an SNS topic. The downside is we'd need to create a topic for each secret or create a registry [CRDs] of sidecars that the controller would notify when the secret changed. We could extend NASE to create an SNS topic. I'll need to think about it.