Automated AWS inventory

Hosts and groups handling

One of the inventory plugin options is the AWS EC2 inventory plugin, which will automatically build inventory:

• https://docs.ansible.com/ansible/latest/collections/amazon/aws/aws_ec2_inventory.html

This is loaded via your ansible.cfg file.

Configuration is done via an aws_ec2.yml file in hosts in your config repo. Here is an example:

plugin: amazon.aws.aws_ec2
filters:
  tag:Ansible: managed
keyed_groups:
  - key: tags.Name
    prefix: ""
  - key: tags.Env
    prefix: ""
  - key: tags.Profile
    prefix: ""
  - key: tags.Infra
    prefix: ""

How it works

The plugin is loading all EC2 instances that are tagged with Ansible: managed and then grouping them by the tags Name, Env, Profile and Infra. Any hyphens in tags are automatically converted to underscores, and the prefixing convention is taken from the default behaviour of the ansible.builtin.constructed plugin, which you can read here - note specifically the leading_separator parameter and its documentation:

• https://docs.ansible.com/ansible/latest/collections/ansible/builtin/constructed_inventory.html#parameter-leading_separator

Extending the example, if you group all infra by the Name tag, effectively your inventory will always contain a group consisting of the name of that machine, prefixed with an underscore, for example the server with the Name tag of dummy-test1-codeenigma-net would end up in a group of one instance like this:

  |--@_dummy_test1_codeenigma_net:
  |  |--ec2-5-21-235-9.eu-west-2.compute.amazonaws.com

In this way we can act on a specific host by invoking its unique group, which is why you see code like this at the top of infrastructure plays where we are loading the target:

- hosts: _dummy_test1_codeenigma_net
  become: true

You could also then start using variables to load the correct hosts at runtime. For example, we have a variable called _aws_resource_name which we pass in via our CI and our generic playbook hosts line looks like this:

- hosts: "_{{ _aws_resource_name | regex_replace('-', '_') }}"
  become: true

Debugging and viewing hosts

You can view the graphed default infrastructure from the command line of a controller with a command like this when logged in as the ce-deploy user, usually deploy:

ansible-inventory -i ~/ce-deploy/hosts/aws_ec2.yml --graph

If you wanted to see the inventory for another boto profile you need to set the AWS_PROFILE environment variable. For example, this would graph the acme profile's inventory:

AWS_PROFILE=acme ansible-inventory -i ~/ce-deploy/hosts/aws_ec2.yml --graph

You will note there are other groupings, for example you can call all the _prod infrastructure because there is also a grouping against the Env tag, or you can call all the _web servers because they are also grouped by Profile, and so on.

Unmanaged infra

If you want a host that is not tagged with Ansible: managed to be "known" to Ansible you need to add it to hosts.yml in your hosts directory.

Using group_vars

Once you understand this, the group_vars directory within this directory starts to make sense. You can set variables that apply to any group that gets created automatically by the inventory plugin, for example, we have a _dummy folder containing variables which will apply to every single server in the dummy infra and take precedence over the defaults in all. Or we can set a _dev folder containing variables for every server tagged in a dev environment, regardless of infra. Ansible will always load the all variables first, then overwrite any variables with other groups as they are discovered, according to precedence.

You could expand on this example config, for example it does not allow mixing infra names and group names but you could easily do this so you have groups like _acme_dev or _acme_web, which could be useful groupings for large infras.