Running Ansible with AWS System Manager

Introduction

Ansible is a fantastic tool to automate tasks and manage fleets of servers

Amazon SSM, or Systems Manager, also offers tools to manage your fleet of servers. One of the tools they offer is Session Manager, which allows you to run an interactive shell without having to open a TCP port to the world or dealing with SSH keys.

Usualy, ansible uses SSH to connect and send payloads to the managed servers, if you're using only SSM to mange your instances, how can Ansible connect to them? Well, by using SSM itself to communicate and an S3 bucket to store the payload.

Debian/Ubuntu users beware

System Manager uses a small script to determine if the connection was sucessfull, by watching the results on Stdout/Stderr. Problem is, they specify /bin/sh as the shell, only to use some Bash specific syntax on it. This is a problem for Debian based distributions, because by default, they symlink /bin/sh to Dash, instead of Bash. Since Dash doesn't support the Bashisms that AWS uses, it will return an error. The best way to fix this, is run a script using SSM to re-link /bin/sh to Bash instead.

As root, run this script on all your Debian instances to change it:

#!/bin/bash
echo "dash dash/sh boolean false" | debconf-set-selections
debconf-show dash
DEBIAN_FRONTEND=noninteractive dpkg-reconfigure -p critical dash

You can do this from AWS SSM Control Pannel by selecting Run Command under Node Management and selecting AWS-RunShellScript as the document. No need to spam sudo on the script, this document runs the scripts as root by default.

After that, all scripts that specify /bin/sh as the interpreter will use Bash instead, which should not cause any more problems.

Preparation

To do that, we need to first make sure all the instances that will be managed can be accessed with SSM. All of them will need an IAM Role with SSM permissions, in addition to that, we need a S3 bucket that the instances can aread from too, so the IAM Role would also need at least read permission on the bucket. This can be acomplished by adding the appropriate permission to the IAM Role attached to the instances or with a Bucket Policy. Make those settings then test them by acessing an instance using SSM, the command for that is aws ssm start-session --target <instance-id>, then use the AWS CLI tool to list the contents of the bucket with the command aws s3 ls s3://ansible-bucket/. If we succeed with the connection and listing the Buclet, we're good to go.

Ansible Configuration

Using the AWS community modules for ANsible, it's possible to get a list of all your EC2 instances and use that as an inventory. To do that, we need to enable it on Ansible's configuration. We can do that by creating an ansible.cfg on our playbook directory. Open the file and add the following lines:

[inventory]
enable_plugins = aws_ec2

Now, let's create the inventory itself. Create an YAML file, like ec2_inventory.yaml with this content:

plugin: aws_ec2
regions:
  - "us-east-1" # replace this with your region

# Example filters. Bring only running instances.
# The commented line show how to filter using a Tag
filters:
  instance-state-name: running
#  tag:OS_Type: Linux 

# Example exclusion. We don't want instances running Windows Server
exclude_filters:
  - platform: 
    - windows

# Building the inventory itself. This example will use the instance-id
# as Hostname. This is needed to use SSM as a cennection method.
# The keyed_groups key will generate groups based on platform details and
# Tags. This is optional.
hostnames:
  - instance-id
keyed_groups:
  - prefix: platform
    key: platform_details
  - prefix: Tags
    key: tags

Now, to test if our inventory is working, we use the command ansible-inventory -i ec2_inventory.yaml --graph. The result should be a tree graph listing all our instances.

@all:
  |--@Tags_Some_Tag:
  |  |--i-12345abcd
  |  |--i-23456cdefg
  |--@Tags_Another_Tag:
  |  |--i-567fghd
  |  |--i-9875poi
  |  |--i-4567kjb
  ...

Running a Playbook

Now lets try running a simple Playbook that colelcts minimum facts and lists the content of /etc/os-release.

---

- name: SSM Playbook
  hosts:
    - all
  gather_facts: false
  vars:
    # This is where we tell Ansible to connect using SSM
    ansible_connection: aws_ssm
    # The bucket that will be used to transfer the payload
    ansible_aws_ssm_bucket_name: ansible-bucket
    # AWS Region where the instances are
    ansible_aws_ssm_region: us-east-1
    # This is why we should use the Instande-Id as hostname, so
    # we can pass it to the SSM module using Ansible's inventory_hostname variable
    ansible_aws_ssm_instance_id: "{{ inventory_hostname }}"
  tasks:
    - name: Collect Facts
      ansible.builtin.setup:
        gather_subset:
          - '!all'
          - '!min'
    - name: Cat a file
      register: os_release
      ansible.builtin.shell: |
        cat /etc/os-release
    - name: Show the content of the file
      debug:
        msg: "os-release: {{os_release}}"

Now, we just watch the output. If everything is correct, we should see the contents of /etc/os-release from all our instances on the terminal.

I hope this mini-guide helps. Any doubts or comments, hit me on Mastodon: @nerdeiro@fosstodon.org

See you on the next one.