Fix userdata for non-custom amis

Author: ChrisMcKee

README.md

@@ -84,6 +84,7 @@ This module supports all 4 OSes, but support for detailed configuration of the n
 are fully supported for Amazon Linux 2 and Windows, and take advantage of the [bootstrap.sh](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh)
 supplied on those AMIs. **NONE** of these inputs are supported on Bottlerocket. On AL2023, only the first 2 are supported.
 
+Note: The fourth option is only available when setting an AMI ID for the node group; otherwise AWS will default to the released AMI for the OS and EKS will append your userdata with the bootstrap script.
 Note that for all OSes, you can supply the complete `userdata` contents, which will be untouched by this module, via `userdata_override_base64`.
 
 
@@ -318,14 +319,14 @@ module "eks_node_group" {
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_additional_tag_map"></a> [additional\_tag\_map](#input\_additional\_tag\_map) | Additional key-value pairs to add to each map in `tags_as_list_of_maps`. Not added to `tags` or `id`.<br/>This is for some rare cases where resources want additional configuration of tags<br/>and therefore take a list of maps with tag key, value, and additional configuration. | `map(string)` | `{}` | no |
-| <a name="input_after_cluster_joining_userdata"></a> [after\_cluster\_joining\_userdata](#input\_after\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node after joining the EKS cluster (after executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
+| <a name="input_after_cluster_joining_userdata"></a> [after\_cluster\_joining\_userdata](#input\_after\_cluster\_joining\_userdata) | Additional (bash for linux, powershell for Windows) commands to execute on each worker node after joining the EKS cluster<br/>(after executing the `bootstrap` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production"<br/>This can only be used with AL2 and Windows AMI types with a custom AMI set as EKS will inject a bootstrapper into all other user-scripts | `list(string)` | `[]` | no |
 | <a name="input_ami_image_id"></a> [ami\_image\_id](#input\_ami\_image\_id) | AMI to use, overriding other AMI specifications, but must match `ami_type`. Ignored if `launch_template_id` is supplied. | `list(string)` | `[]` | no |
 | <a name="input_ami_release_version"></a> [ami\_release\_version](#input\_ami\_release\_version) | The EKS AMI "release version" to use. Defaults to the latest recommended version.<br/>For Amazon Linux, it is the "Release version" from [Amazon AMI Releases](https://github.com/awslabs/amazon-eks-ami/releases)<br/>For Bottlerocket, it is the release tag from [Bottlerocket Releases](https://github.com/bottlerocket-os/bottlerocket/releases) without the "v" prefix.<br/>For Windows, it is "AMI version" from [AWS docs](https://docs.aws.amazon.com/eks/latest/userguide/eks-ami-versions-windows.html).<br/>Note that unlike AMI names, release versions never include the "v" prefix.<br/>Examples:<br/>  AL2: 1.29.3-20240531<br/>  Bottlerocket: 1.2.0 or 1.2.0-ccf1b754<br/>  Windows: 1.29-2024.04.09 | `list(string)` | `[]` | no |
 | <a name="input_ami_type"></a> [ami\_type](#input\_ami\_type) | Type of Amazon Machine Image (AMI) associated with the EKS Node Group.<br/>Defaults to `AL2_x86_64`. Valid values: `AL2_x86_64, AL2_x86_64_GPU, AL2_ARM_64, CUSTOM, BOTTLEROCKET_ARM_64, BOTTLEROCKET_x86_64, BOTTLEROCKET_ARM_64_FIPS, BOTTLEROCKET_x86_64_FIPS, BOTTLEROCKET_ARM_64_NVIDIA, BOTTLEROCKET_x86_64_NVIDIA, WINDOWS_CORE_2019_x86_64, WINDOWS_FULL_2019_x86_64, WINDOWS_CORE_2022_x86_64, WINDOWS_FULL_2022_x86_64, AL2023_x86_64_STANDARD, AL2023_ARM_64_STANDARD, AL2023_x86_64_NEURON, AL2023_x86_64_NVIDIA`. | `string` | `"AL2_x86_64"` | no |
 | <a name="input_associate_cluster_security_group"></a> [associate\_cluster\_security\_group](#input\_associate\_cluster\_security\_group) | When true, associate the default cluster security group to the nodes. If disabled the EKS managed security group will not<br/>be associated to the nodes and you will need to provide another security group that allows the nodes to communicate with<br/>the EKS control plane. Be aware that if no `associated_security_group_ids` or `ssh_access_security_group_ids` are provided,<br/>then the nodes will have no inbound or outbound rules. | `bool` | `true` | no |
 | <a name="input_associated_security_group_ids"></a> [associated\_security\_group\_ids](#input\_associated\_security\_group\_ids) | A list of IDs of Security Groups to associate the node group with, in addition to the EKS' created security group.<br/>These security groups will not be modified. | `list(string)` | `[]` | no |
 | <a name="input_attributes"></a> [attributes](#input\_attributes) | ID element. Additional attributes (e.g. `workers` or `cluster`) to add to `id`,<br/>in the order they appear in the list. New attributes are appended to the<br/>end of the list. The elements of the list are joined by the `delimiter`<br/>and treated as a single ID element. | `list(string)` | `[]` | no |
-| <a name="input_before_cluster_joining_userdata"></a> [before\_cluster\_joining\_userdata](#input\_before\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node before joining the EKS cluster (before executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
+| <a name="input_before_cluster_joining_userdata"></a> [before\_cluster\_joining\_userdata](#input\_before\_cluster\_joining\_userdata) | Additional `bash` commands to execute on each worker node before joining the EKS cluster (before executing the `bootstrap.sh/ps1` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production | `list(string)` | `[]` | no |
 | <a name="input_block_device_map"></a> [block\_device\_map](#input\_block\_device\_map) | Map of block device name specification, see [launch\_template.block-devices](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template#block-devices). | <pre>map(object({<br/>    no_device    = optional(bool, null)<br/>    virtual_name = optional(string, null)<br/>    ebs = optional(object({<br/>      delete_on_termination = optional(bool, true)<br/>      encrypted             = optional(bool, true)<br/>      iops                  = optional(number, null)<br/>      kms_key_id            = optional(string, null)<br/>      snapshot_id           = optional(string, null)<br/>      throughput            = optional(number, null)<br/>      volume_size           = optional(number, 20)<br/>      volume_type           = optional(string, "gp3")<br/>    }))<br/>  }))</pre> | <pre>{<br/>  "/dev/xvda": {<br/>    "ebs": {}<br/>  }<br/>}</pre> | no |
 | <a name="input_block_device_mappings"></a> [block\_device\_mappings](#input\_block\_device\_mappings) | DEPRECATED: Use `block_device_map` instead.<br/>List of block device mappings for the launch template.<br/>Each list element is an object with a `device_name` key and<br/>any keys supported by the `ebs` block of `launch_template`. | `list(any)` | `null` | no |
 | <a name="input_bootstrap_additional_options"></a> [bootstrap\_additional\_options](#input\_bootstrap\_additional\_options) | Additional options to bootstrap.sh. DO NOT include `--kubelet-additional-args`, use `kubelet_additional_options` var instead. Not used with AL2023 AMI types. | `list(string)` | `[]` | no |

README.yaml

@@ -101,6 +101,7 @@ introduction: |-
   are fully supported for Amazon Linux 2 and Windows, and take advantage of the [bootstrap.sh](https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh)
   supplied on those AMIs. **NONE** of these inputs are supported on Bottlerocket. On AL2023, only the first 2 are supported.
 
+  Note: The fourth option is only available when setting an AMI ID for the node group; otherwise AWS will default to the released AMI for the OS and EKS will append your userdata with the bootstrap script.
   Note that for all OSes, you can supply the complete `userdata` contents, which will be untouched by this module, via `userdata_override_base64`.
 
 

launch-template.tf

@@ -197,6 +197,10 @@ resource "aws_launch_template" "default" {
       condition     = contains(["AL2", "WINDOWS"], local.ami_os) || length(local.userdata_vars.after_cluster_joining_userdata) == 0 || (local.ami_os == "AL2" || local.ami_os == "WINDOWS")
       error_message = format("The input `after_cluster_joining_userdata` is not supported for %v.", title(lower(local.ami_os)))
     }
+    precondition {
+      condition     = length(local.userdata_vars.after_cluster_joining_userdata) == 0 || length(var.ami_image_id) != 0 && length(local.userdata_vars.after_cluster_joining_userdata) > 0 && (local.ami_os == "AL2" || local.ami_os == "WINDOWS")
+      error_message = format("The input `after_cluster_joining_userdata` is not supported for %v, a custom ami_image_id must be set for this functionality", title(lower(local.ami_os)))
+    }
   }
 }
 

userdata.tf

@@ -49,7 +49,11 @@ locals {
     WINDOWS      = "${path.module}/userdata_nt.tpl"
   }
 
-
+  userdata_bootstrapper_template_file = {
+    AL2          = "${path.module}/userdata_bootstrap.tpl"
+    BOTTLEROCKET = "${path.module}/userdata_bootstrap.tpl"
+    WINDOWS      = "${path.module}/userdata_bootstrap_nt.tpl"
+  }
 
   # When suppressing EKS bootstrap, add --register-with-taints to kubelet_extra_args,
   #   e.g. --register-with-taints=test=:PreferNoSchedule
@@ -71,6 +75,7 @@ locals {
   kubelet_extra_args_yaml = replace(local.kubelet_extra_args, "--", "\n      - >-\n        --")
 
   userdata_vars = {
+    bootstrap_script                = (length(var.ami_image_id) > 0 && length(var.after_cluster_joining_userdata) > 0 && (local.ami_os == "AL2" || local.ami_os == "WINDOWS")) ? file(local.userdata_template_file[local.ami_os]) : ""
     before_cluster_joining_userdata = length(var.before_cluster_joining_userdata) == 0 ? "" : join("\n", var.before_cluster_joining_userdata)
     kubelet_extra_args              = local.kubelet_extra_args
     kubelet_extra_args_yaml         = local.kubelet_extra_args_yaml

userdata.tpl

@@ -15,7 +15,7 @@ export KUBELET_EXTRA_ARGS="${kubelet_extra_args}"
 %{ endif }
 %{ if length(kubelet_extra_args) > 0 || length (bootstrap_extra_args) > 0 || length (after_cluster_joining_userdata) > 0 }
 
-/etc/eks/bootstrap.sh --apiserver-endpoint '${cluster_endpoint}' --b64-cluster-ca '${certificate_authority_data}' ${bootstrap_extra_args} '${cluster_name}'
+${bootstrap_script}
 
 ${after_cluster_joining_userdata}
 %{ endif }

userdata_bootstrap.tpl

@@ -0,0 +1 @@
+/etc/eks/bootstrap.sh --apiserver-endpoint '${cluster_endpoint}' --b64-cluster-ca '${certificate_authority_data}' ${bootstrap_extra_args} '${cluster_name}'

userdata_bootstrap_nt.tpl

@@ -0,0 +1,5 @@
+[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
+[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
+[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
+
+& $EKSBootstrapScriptFile -EKSClusterName "${cluster_name}" -APIServerEndpoint "${cluster_endpoint}" -Base64ClusterCA "${certificate_authority_data}" ${bootstrap_extra_args} -KubeletExtraArgs "${kubelet_extra_args}" 3>&1 4>&1 5>&1 6>&1

userdata_nt.tpl

@@ -26,4 +26,13 @@ if ($disks_to_adjust -ne $null) {
 [string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
 [string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
 
+${bootstrap_script}
+
+try{
+${after_cluster_joining_userdata}
+}catch{
+  Write-Host "An error occurred in post-script" -ForegroundColor Red
+  Write-Host $_.ScriptStackTrace
+}
+
 </powershell>

variables.tf

@@ -358,14 +358,18 @@ variable "resources_to_tag" {
 
 variable "before_cluster_joining_userdata" {
   type        = list(string)
-  description = "Additional `bash` commands to execute on each worker node before joining the EKS cluster (before executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production"
+  description = "Additional `bash` commands to execute on each worker node before joining the EKS cluster (before executing the `bootstrap.sh/ps1` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production"
   default     = []
   nullable    = false
 }
 
 variable "after_cluster_joining_userdata" {
   type        = list(string)
-  description = "Additional `bash` commands to execute on each worker node after joining the EKS cluster (after executing the `bootstrap.sh` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production"
+  description = <<-EOT
+    Additional (bash for linux, powershell for Windows) commands to execute on each worker node after joining the EKS cluster
+    (after executing the `bootstrap` script). For more info, see https://kubedex.com/90-days-of-aws-eks-in-production"
+    This can only be used with AL2 and Windows AMI types with a custom AMI set as EKS will inject a bootstrapper into all other user-scripts
+    EOT
   default     = []
   nullable    = false
 }