devops

How to filter, join and map lists in Ansible

TL;DR

    • use “select” filter to filter a list and “match” to combine it with reg exps, like:
      "{{ ansible_interfaces | select('match', '^(eth|wlan)[0-9]+') | list }}"
    • use “map” to map list elements, like:
      "{{ ansible_interfaces | map('upper') | list }}"
    • use just “+” operator to combine two lists into one, like:
      "{{ ansible_interfaces + [\"VETH-1\", \"VETH-2\"] }}"

Ansible filters and lists operators

Ansible provides a rich set of filters, which you can apply to your variables. Recently, I needed to filter and map a list of host interfaces. I couldn’t remind myself how to do this, so I jumped to ansible filters docs. But I couldn’t find it there. Well, you have to remember, that expression evaluation in Ansible is based on Jinja2, so you need to check Jinja2 filters documentation as well. You will find basic operators there.

When operating on lists, functional languages are a great example: by combining just a few list operators, you can get almost any desired transformation of the list. These basic operators include:

  • filter – to select just matching elements from the list and create a new list from them
  • map – to convert all elements on the list according to transformation function that operates on a single element
  • reduce – to convert (aggregate) all elements of a list into a single value
  • flat map – to merge multiple lists into a single list

Let’s now see how to perform them in Ansible 2.4.

Our example – list all network interfaces

As our example list we want to transform in ansible, let’s use a list of local network interfaces that Ansible discovers during setup and stores in “ansible_interfaces” list. A simple play that just fetches and prints the list can look like this:

- hosts: all
  tasks:
    - name: print interfaces
      debug:
        msg: "{{ ansible_interfaces }}"

Let’s save the above code in file ‘list_operators.yml’ and run it with:

ansible-playbook -c local -i 'localhost,' list_operators.yml

On my PC this produces the following output:

PLAY [all] ****************************************************************

TASK [Gathering Facts] ****************************************************
ok: [localhost]

TASK [print interfaces] ***************************************************
ok: [localhost] {
    "msg": [
        "docker0", 
        "lo", 
        "veth3e8c318", 
        "wlan0",
        "eth0", 
        "vethe2459c9"
    ]
}

PLAY RECAP ****************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

Filter a list in Ansible

Let’s assume we want to filter the interfaces list to include only “eth” or “wlan” types of interfaces. As you can find here in Jinja2 docs, we can use the “select” filter to filter the list. But the great thing – and not that easy to find – is that we can combine “select” with “match” operator to test for matching against a regular expression! This brings us to the following solution:

- hosts: all
  tasks:
    - name: print interfaces
      debug:
        msg: "{{ ansible_interfaces | select('match', '^(eth|wlan)[0-9]+') | list }}"

So, what it does is that it passes each element on the list “ansible_interfaces” to a “match” test. This test checks with the regexp if the element starts with “eth” or “wlan” followed by at least 1 digit. Elements that match are combined back again into a list with the “list” filter. The code above produces the following output:

TASK [print interfaces] ***************************************************
ok: [localhost] {
    "msg": [
        "wlan0", 
        "eth0"
    ]
}

Map list elements in Ansible

OK, let’s now see how we can use the map filter in Ansible. Let’s assume that we want to convert each interface name on our list to upper case only. Jinja2 has a filter for that, so our Ansible task becomes:

- hosts: all
  tasks:
    - name: print interfaces
      debug:
        msg: "{{ ansible_interfaces | map('upper') | list }}"

and produces the following output:

TASK [print interfaces] ***************************************************
ok: [localhost] {
    "msg": [
        "DOCKER0", 
        "LO", 
        "VETH3E8C318", 
        "WLAN0", 
        "DOCKERCON", 
        "ETH0", 
        "VETHE2459C9"
    ]
}

Merge two lists into one in Ansible

Now, what if we want to merge two lists into a single one that contains elements of both of them? That’s really easy, as the normal “+” operator works for lists and does exactly that. So, running this code:

- hosts: all
  tasks:
    - name: print interfaces
      debug:
        msg: "{{ ansible_interfaces + [\"VETH-1\", \"VETH-2\"] }}"

produces the following output, where the list of local interfaces is extended with elements “VETH-1” and “VETH-2” from the second list:

TASK [print interfaces] ***************************************************
ok: [localhost] {
    "msg": [
        "docker0", 
        "lo", 
        "veth3e8c318", 
        "wlan0", 
        "dockercon", 
        "eth0", 
        "vethe2459c9", 
        "VETH-1", 
        "VETH-2"
    ]
}

Aggregate / reduce

Here the problem is trickier – it depends on what you really want to do with list’s elements. Probably the most popular aggregation filter is “join”, which just joins list elements into a single stream with elements separated by a given separator. There are a few more, but in general, when you need something more complicated, you either have to escape to a full Jinja2 template and do something custom in a “{% for %}” loop or implement your own filter.

Related

This article also has a continuation: Advanced list operations in Ansible