TL;DR

• “selectattr” filter in Ansible is useful for filtering lists based on attributes of the objects in the list

  •  selectattr('name', 'match', 'eth[2-9]') 

• Ansible’s “sum” filter can be used for reducing lists into new objects (including lists)

  • sum(attribute='ips', start=[]) 

Advanced list operations in Ansible

Intro

In the previous entry, I gave you an overview of how you can do all the basic operations on lists in Ansible. Soon after that, I faced a problem, that was still solvable without writing an Ansible plugin. Using list filters was enough, yet the task definitely wasn’t simple. Let me show you what the challenge was and how you can combine some advanced list filters in Ansible to perform more complicated list tasks.

The problem

In our company, we use Ansible to configure some multi-interface multi-IP networking instances running in AWS. These instances are used to manage client’s IPs, so we need to have full control over which IP is assigned where. To deal with that, we have an Ansible role that reads a YAML file in the following format for each of instance’s interfaces:

# file host_vars/HOST/interfaces-eth1.yml
interfaces_eth1:
- gw: 10.10.10.1
  name: eth1
  ips:
  - {ip: 10.10.10.104, owner: TEST, project: The Project}
  - {ip: 10.10.10.105, owner: Our Client, project: The Project} 

 
# file host_vars/HOST/interfaces-eth2.yml 
interfaces_eth2: 
- gw: 10.10.10.1 ips:
  name: eth2
  - {ip: 10.10.10.204, owner: TEST, project: The Project}
  - {ip: 10.10.10.205, owner: Our Client, project: The Project}
  - {ip: 10.10.10.206, owner: Our Client, project: The Project}
  - {ip: 10.10.10.207, owner: Our Client, project: The Project}
  - {ip: 10.10.10.208, owner: Our Client, project: The Project}
  - {ip: 10.10.10.209, owner: Our Client, project: The Project}

Then, we combine all per-interface configs into a single config file for an instance like this:

# file host_vars/HOST/interfaces.yml
interfaces: "{{ interfaces_eth1 }} + {{ interfaces_eth2 }}"

Now, the problem we were facing is: on each instance, we have at least 1 IP address dedicated to monitoring and checking if the given interface is up and running. Such IP address has the owner field set to “TEST”. So, the challenge is to find at least one “TEST” IP on each instance. Basically, we want the first “TEST” IP address out of all addresses assigned to all interfaces of any single networking machine.  Additionally, we have to ignore “eth0” and “eth1” interfaces, as they are being used for other purposes.

The solution, Ansible and Jinja2 way

If you want to test the solution on your local host, create 3 files in the directory ‘host_vars/localhost’ as above. The problem can be solved with the following ansible code. Let’s first have a look and then I will explain it step by step:

# file playbook.yml
- hosts: all
 tasks:
 - set_fact:
 test_ip: "{{ interfaces 
 | selectattr('name', 'match', 'eth[2-9]') 
 | sum(attribute='ips', start=[]) 
 | selectattr('owner', 'equalto', 'TEST') 
 | map(attribute='ip') 
 | list 
 | first 
 | default('NOT_FOUND') }}"
 - debug:
 msg: "The TEST IP is {{ test_ip }}"

In line 5, we start with our list of interface configurations, created from both “interfaces_eth1” and “interfaces_eth2”, which are one-element lists.

Line 6 uses “selectattr” filter, which acts as a filter for a list of objects. This filter passes only objects matching certain criteria. In our case, the two-element list of interface configuration blocks is checked one element at a time and only blocks having attribute “name” matching a regular expression “eth[2-9]” are passed forward. The fact that we want to match by a regular expression is reflected by the 2nd argument set to “match”.

Then, line 7 does a nice trick: it’s a reduction operator that acts on all elements of a list and produces a single resulting object. Its arguments are object’s attribute to act on and the initial state. The operand applied here for reduction is of course “sum”. So, we start with an empty list (the 2nd argument) and then for each interface definition block on our list, we add to the empty list the attribute “ips” of the interface block. The result is a single list of all IP addresses assigned to all interfaces that were not filtered out by the previous line.

Next, in line 8, from all the IPs on the list, we select only ones that have the attribute “owner” equal to “TEST”. Now, things get simpler. We want only the IP address, without “owner” and ” project” attributes, so we “map” every object into a new one by selecting only the attribute “ip”. In line 9 we turn the sequence of IPs into a list again. In line 10 we select just the first element of the list, in case we have found more than one. If we found none, we add “default” filter at the very end.

OK, let’s run it.

$ ansible-playbook -c local -i 'localhost,' playbook.yml

PLAY [all] *********************************************************

TASK [Gathering Facts] *********************************************
ok: [localhost]

TASK [set_fact] ****************************************************
ok: [localhost]

TASK [debug] *******************************************************
ok: [localhost] {
 "msg": "The TEST IP is 10.10.10.204"
}

PLAY RECAP *********************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0 

And it works! Tested with Ansible 2.4.3.0, Jinja2 2.9.6 and python 2.7.14.