Skip to content

Commit fa35aaa

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #4359 from praddy26/fix-4019-renew-cert
fix: Race condition in webhook certificate renewal with cert-manager self-signed issuer without a dedicated CA certificate
2 parents 584e3fb + 44ca046 commit fa35aaa

File tree

4 files changed

+174
-26
lines changed

4 files changed

+174
-26
lines changed

‎docs/deploy/cert-manager.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Cert Manager Integration
2+
3+
The AWS Load Balancer Controller uses admission webhooks to validate and mutate resources. These webhooks require TLS certificates to operate securely. You can use cert-manager to automatically provision and manage these certificates.
4+
5+
## Upgrade Notes
6+
7+
When upgrading from a previous version, the following scenarios are handled automatically:
8+
9+
1. If you have existing TLS secrets and `keepTLSSecret: true` (default):
10+
- Existing secrets are preserved
11+
- No new certificates are created
12+
- Your existing certificate setup continues to work as before
13+
14+
2. If you're using cert-manager with a custom issuer:
15+
- Set `certManager.issuerRef` to keep using your issuer
16+
- The new CA hierarchy will not be created
17+
- Your existing certificate configuration is preserved
18+
19+
3. If you're using cert-manager without a custom issuer:
20+
- A new CA hierarchy will be created
21+
- New certificates will be issued using this CA
22+
- The transition is handled automatically by cert-manager
23+
24+
## How it Works
25+
26+
When using cert-manager integration, the controller creates a certificate hierarchy that consists of:
27+
28+
1. A self-signed issuer used only to create the root CA certificate
29+
2. A root CA certificate with a 5-year validity period
30+
3. A CA issuer that uses the root certificate to sign webhook serving certificates
31+
4. Webhook serving certificates with 1-year validity that are automatically renewed
32+
33+
This setup prevents race conditions during certificate renewal by:
34+
- Using a long-lived (5 years) root CA certificate that remains stable
35+
- Only renewing the serving certificates while keeping the CA constant
36+
- Letting cert-manager's CA injector handle caBundle updates in webhook configurations
37+
38+
## Configuration
39+
40+
To enable cert-manager integration, set `enableCertManager: true` in your Helm values.
41+
42+
You can customize the certificate configuration through these values:
43+
44+
```yaml
45+
enableCertManager: true
46+
47+
certManager:
48+
# Webhook serving certificate configuration
49+
duration: "8760h0m0s" # 1 year (default)
50+
renewBefore: "720h0m0s" # 30 days (optional)
51+
revisionHistoryLimit: 10 # Optional
52+
53+
# Root CA certificate configuration
54+
rootCert:
55+
duration: "43800h0m0s" # 5 years (default)
56+
57+
# Optional: Use your own issuer instead of the auto-generated one
58+
# issuerRef:
59+
# name: my-issuer
60+
# kind: ClusterIssuer
61+
```
62+
63+
### Using Custom Issuers
64+
65+
If you want to use your own cert-manager issuer instead of the auto-generated CA, you can configure it through `certManager.issuerRef`:
66+
67+
```yaml
68+
certManager:
69+
issuerRef:
70+
name: my-issuer
71+
kind: ClusterIssuer # or Issuer
72+
```
73+
74+
When a custom issuer is specified:
75+
- The controller will not create its own CA certificate chain
76+
- The specified issuer will be used directly to issue webhook serving certificates
77+
- You are responsible for ensuring the issuer is properly configured and available
78+
79+
### Certificate Renewal
80+
81+
1. Root CA Certificate:
82+
- Valid for 5 years by default
83+
- Used only for signing webhook certificates
84+
- Not renewed automatically to maintain stability
85+
86+
2. Webhook Serving Certificates:
87+
- Valid for 1 year by default
88+
- Renewed automatically 30 days before expiry
89+
- Updates handled seamlessly by cert-manager
90+
91+
### Best Practices
92+
93+
1. Use the default certificate hierarchy unless you have specific requirements
94+
2. If using a custom issuer, ensure it's highly available and properly configured
95+
3. Monitor certificate resources for renewal status and potential issues
96+
4. Keep cert-manager up to date to benefit from the latest improvements

‎helm/aws-load-balancer-controller/templates/cert-manager.yaml‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
{{- if and .Values.enableCertManager (not .Values.certManager.issuerRef) -}}
2+
# Create a selfsigned Issuer, in order to create a root CA certificate for
3+
# signing webhook serving certificates
4+
apiVersion: cert-manager.io/v1
5+
kind: Issuer
6+
metadata:
7+
name: {{ template "aws-load-balancer-controller.namePrefix" . }}-selfsigned-issuer
8+
namespace: {{ .Release.Namespace }}
9+
labels:
10+
{{- include "aws-load-balancer-controller.labels" . | nindent 4 }}
11+
spec:
12+
selfSigned: {}
13+
---
14+
# Generate a CA Certificate used to sign certificates for the webhook
15+
apiVersion: cert-manager.io/v1
16+
kind: Certificate
17+
metadata:
18+
name: {{ template "aws-load-balancer-controller.namePrefix" . }}-root-cert
19+
namespace: {{ .Release.Namespace }}
20+
labels:
21+
{{- include "aws-load-balancer-controller.labels" . | nindent 4 }}
22+
spec:
23+
secretName: {{ template "aws-load-balancer-controller.namePrefix" . }}-root-cert
24+
duration: {{ .Values.certManager.rootCert.duration | default "43800h0m0s" | quote }}
25+
issuerRef:
26+
name: {{ template "aws-load-balancer-controller.namePrefix" . }}-selfsigned-issuer
27+
commonName: "ca.webhook.aws-load-balancer-controller"
28+
isCA: true
29+
subject:
30+
organizations:
31+
- aws-load-balancer-controller
32+
---
33+
# Create an Issuer that uses the above generated CA certificate to issue certs
34+
apiVersion: cert-manager.io/v1
35+
kind: Issuer
36+
metadata:
37+
name: {{ template "aws-load-balancer-controller.namePrefix" . }}-root-issuer
38+
namespace: {{ .Release.Namespace }}
39+
labels:
40+
{{- include "aws-load-balancer-controller.labels" . | nindent 4 }}
41+
spec:
42+
ca:
43+
secretName: {{ template "aws-load-balancer-controller.namePrefix" . }}-root-cert
44+
{{- end -}}

‎helm/aws-load-balancer-controller/templates/webhook.yaml‎

Lines changed: 22 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,9 @@ metadata:
1212
{{- include "aws-load-balancer-controller.labels" . | nindent 4 }}
1313
webhooks:
1414
- clientConfig:
15-
{{ if not $.Values.enableCertManager -}}
15+
{{- if not $.Values.enableCertManager }}
1616
caBundle: {{ $tls.caCert }}
17-
{{ end }}
17+
{{- end }}
1818
service:
1919
name: {{ template "aws-load-balancer-controller.webhookService" . }}
2020
namespace: {{ $.Release.Namespace }}
@@ -58,9 +58,9 @@ webhooks:
5858
sideEffects: None
5959
{{- if .Values.enableServiceMutatorWebhook }}
6060
- clientConfig:
61-
{{ if not $.Values.enableCertManager -}}
61+
{{- if not $.Values.enableCertManager }}
6262
caBundle: {{ $tls.caCert }}
63-
{{ end }}
63+
{{- end }}
6464
service:
6565
name: {{ template "aws-load-balancer-controller.webhookService" . }}
6666
namespace: {{ $.Release.Namespace }}
@@ -95,9 +95,9 @@ webhooks:
9595
sideEffects: None
9696
{{- end }}
9797
- clientConfig:
98-
{{ if not $.Values.enableCertManager -}}
98+
{{- if not $.Values.enableCertManager }}
9999
caBundle: {{ $tls.caCert }}
100-
{{ end }}
100+
{{- end }}
101101
service:
102102
name: {{ template "aws-load-balancer-controller.webhookService" . }}
103103
namespace: {{ $.Release.Namespace }}
@@ -130,9 +130,9 @@ metadata:
130130
{{- include "aws-load-balancer-controller.labels" . | nindent 4 }}
131131
webhooks:
132132
- clientConfig:
133-
{{ if not $.Values.enableCertManager -}}
133+
{{- if not $.Values.enableCertManager }}
134134
caBundle: {{ $tls.caCert }}
135-
{{ end }}
135+
{{- end }}
136136
service:
137137
name: {{ template "aws-load-balancer-controller.webhookService" . }}
138138
namespace: {{ $.Release.Namespace }}
@@ -159,9 +159,9 @@ webhooks:
159159
- ingressclassparams
160160
sideEffects: None
161161
- clientConfig:
162-
{{ if not $.Values.enableCertManager -}}
162+
{{- if not $.Values.enableCertManager }}
163163
caBundle: {{ $tls.caCert }}
164-
{{ end }}
164+
{{- end }}
165165
service:
166166
name: {{ template "aws-load-balancer-controller.webhookService" . }}
167167
namespace: {{ $.Release.Namespace }}
@@ -183,9 +183,9 @@ webhooks:
183183
sideEffects: None
184184
{{- if not $.Values.webhookConfig.disableIngressValidation }}
185185
- clientConfig:
186-
{{ if not $.Values.enableCertManager -}}
186+
{{- if not $.Values.enableCertManager }}
187187
caBundle: {{ $tls.caCert }}
188-
{{ end }}
188+
{{- end }}
189189
service:
190190
name: {{ template "aws-load-balancer-controller.webhookService" . }}
191191
namespace: {{ $.Release.Namespace }}
@@ -222,6 +222,9 @@ data:
222222
tls.crt: {{ $tls.clientCert }}
223223
tls.key: {{ $tls.clientKey }}
224224
{{- else }}
225+
{{- $secretName := (include "aws-load-balancer-controller.webhookCertSecret" .) -}}
226+
{{- $secret := lookup "v1" "Secret" .Release.Namespace $secretName -}}
227+
{{- if not (and .Values.keepTLSSecret $secret) }}
225228
apiVersion: cert-manager.io/v1
226229
kind: Certificate
227230
metadata:
@@ -234,12 +237,16 @@ spec:
234237
- {{ template "aws-load-balancer-controller.webhookService" . }}.{{ .Release.Namespace }}.svc
235238
- {{ template "aws-load-balancer-controller.webhookService" . }}.{{ .Release.Namespace }}.svc.{{ .Values.cluster.dnsDomain }}
236239
issuerRef:
240+
{{- if .Values.certManager.issuerRef }}
241+
{{- toYaml .Values.certManager.issuerRef | nindent 4 }}
242+
{{- else }}
237243
kind: Issuer
238-
name: {{ template "aws-load-balancer-controller.namePrefix" . }}-selfsigned-issuer
244+
name: {{ template "aws-load-balancer-controller.namePrefix" . }}-root-issuer
245+
{{- end }}
239246
secretName: {{ template "aws-load-balancer-controller.webhookCertSecret" . }}
240247
{{- with .Values.certManager -}}
241248
{{ if .duration }}
242-
duration: {{ .duration }}
249+
duration: {{ .duration | default "8760h0m0s" | quote }}
243250
{{- end }}
244251
{{- if .renewBefore }}
245252
renewBefore: {{ .renewBefore }}
@@ -248,14 +255,5 @@ spec:
248255
revisionHistoryLimit: {{ .revisionHistoryLimit }}
249256
{{- end }}
250257
{{- end }}
251-
---
252-
apiVersion: cert-manager.io/v1
253-
kind: Issuer
254-
metadata:
255-
name: {{ template "aws-load-balancer-controller.namePrefix" . }}-selfsigned-issuer
256-
namespace: {{ .Release.Namespace }}
257-
labels:
258-
{{ include "aws-load-balancer-controller.labels" . | indent 4 }}
259-
spec:
260-
selfSigned: {}
258+
{{- end }}
261259
{{- end }}

‎helm/aws-load-balancer-controller/values.yaml‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -117,10 +117,20 @@ enableCertManager: false
117117

118118
# Overrideable variables when enableCertManager is set to true
119119
certManager:
120-
duration:
121-
renewBefore:
120+
# Webhook serving certificate configuration
121+
duration: "8760h0m0s" # 1 year
122+
renewBefore: "720h0m0s" # 30 days
122123
revisionHistoryLimit:
123124

125+
# Root CA certificate configuration
126+
rootCert:
127+
duration: "43800h0m0s" # 5 years
128+
129+
# Optional: custom issuer reference
130+
# issuerRef:
131+
# name: my-issuer
132+
# kind: ClusterIssuer
133+
124134
# The name of the Kubernetes cluster. A non-empty value is required
125135
clusterName:
126136

0 commit comments

Comments
 (0)