For some time now, I’m also doing a project that anyone can use to create your own Kubernetes cluster, that is entirely driven by the Flux project. What’s probably the best part of it, by using a free layer infrastructure from Oracle Cloud, you can run this cluster completely for free! So this gives you a fully functional Kubernetes cluster, managed by a great GitOps tool, and without paying a penny. Check the repo here: https://github.com/piontec/free-oci-kubernetes/

The post How to Get a Fully Functional, GitOps Driven Kubernetes Cluster for Free appeared first on Tailored Cloud.

The post Cloud-native observability series: season finale available now! appeared first on Tailored Cloud.

In part 2, I’m discussing how to build a simple micro services app that will be used as a ‘lab rat’ for our experiments with the cloud-native stack later.

In part 3, I’m describing how you can deploy your application (just a binary) as a full fledged Kubernetes concept, up to the point where we can pack everything into a Helm chart.

Enjoy!

The post Building applications for the cloud-native stack – continuation. appeared first on Tailored Cloud.

It is a year since a wrote a blog post here. A lot has changed over this year in my professional life. After quitting my previous job, I was looking for new opportunities and eventually decided to join the cool team at Giant Swarm as a platform architect. I have to say this was a great choice! My focus is now on enabling our customers to run applications on top of Kubernetes clusters.

Still, I didn’t want to stop blogging. Fortunately, I can now connect everything as part of my new job! So, I’m switching my focus slightly to cloud-native applications running on Kubernetes than just Kubernetes. This is also the topic of a blog series called “A Few Dozen Lines of Code”, the first part of which was just published on our company blog!

This series is meant to show you how using an opinionated cloud-native stack of tools installed on your Kubernetes cluster can make your dev and operations team happier than ever Stay tuned for next parts, they should come shortly.

So, this entry is kind of “Schrödinger’s entry”. It is a new entry and it is not – as I’m just pointing you to my article on the company blog. Still, I will inform you if anything written by me is published there. Meanwhile, you can check other engineering articles published there, it’s good content.

The post Schrödinger’s come back to blogging: using the cloud-native stack series. appeared first on Tailored Cloud.

• SmartNat is a Kubernetes ingress controller for exposing a massive number of TCP/UDP services to the outside world using just 1 server
• available on GitHub: https://github.com/DevFactory/smartnat

SmartNat – Kubernetes ingress controller for TCP/UDP services

Some time ago I wrote posts about writing a very simple Kubernetes controller and using operator framework to create a more complete one. Well, at the same time, I was starting to work on one at the company I’m working for. Meanwhile, I convinced management at the company to release the project as an OpenSource – and here it is!

The project is called SmartNat. It’s a Kubernetes ingress controller for TCP/UDP services that allows you to drive external traffic to your Services. It’s kind of Service with NodePort, but on strong steroids. It runs on a separate instance (well, you can run however you like, but this makes the most sense) and interconnects external (usually public) network with the subnet used by your Kubernetes cluster. SmartNat allows you to use multiple network interfaces, each one having multiple IP addresses to forward traffic from an external network to your services on a port-by-port basis. That way using just a single server or instance you can easily expose hundreds or even thousands of Services. The important property is that all of this is done using L3/L4 tools only, so SmartNat helps where HTTP based Ingresses can’t. Additionally, SmartNat supports simple traffic filtering of traffic coming from external subnet and also HA mode.

If you’re interested, check the project on github: https://github.com/DevFactory/smartnat.

The post SmartNat – dirt cheap Kubernetes ingress controller for TCP/UDP services appeared first on Tailored Cloud.

• Set “TF_PLUGIN_CACHE_DIR”  environment variable to an empty dir, then rerun “terraform init” to switch to a shared providers directory. This allows to cache Terraform providers.

Dynamic Terraform providers

Have you noticed, that when you run “terraform init”, Terraform fetches additional binaries from the Internet? These binaries are called “dynamic providers” and they are (mainly) interfaces to the infrastructure operators you’re using in your Terraform code.

So, for example, when you’re using AWS as an infrastructure provider, and you also use some templating to generate config files, you might run “terraform version” and see something like this:

$ terraform version
Terraform v0.11.7
+ provider.aws v1.20.0
+ provider.null v1.0.0
+ provider.template v1.0.0

In the example above, we have 3 providers being used: the “null” and “template” provider and a specialized provider for talking with AWS API. You can find a list of projects developing these providers on Github. But where does Terraform keep these additional provider binaries? Well, each directory that is used as a root of Terraform project (where you run “terraform init”) has a hidden “.terraform” directory. There you can find the downloaded providers:

ll -h .terraform/plugins/linux_amd64
-rwxrwxr-x 1 piontec piontec 239 May 24 06:28 lock.json
-rwxr-xr-x 1 piontec piontec 71M May 24 06:28 terraform-provider-aws_v1.20.0_x4
-rwxr-xr-x 1 piontec piontec 12M Apr 19 07:29 terraform-provider-null_v1.0.0_x4
-rwxr-xr-x 1 piontec piontec 12M Apr 19 07:29 terraform-provider-template_v1.0.0_x4

This approach is nice, as all your projects are independent and self-contained: you can copy the whole directory to some other place or even machine and everything will still work. However, this approach has also a drawback. If you have multiple Terraform working directories, you have to download and store your providers for every one of them. As you can see, the basic set of providers listed above takes around 100MB – not much for today standards, but still. This gets even worse when you have multiple Terraform projects used by a group of people.  And when every one of them runs “terraform init”, he or she needs to wait for the binaries to be downloaded.

A cache directory for providers to the rescue

The solution to the above problem is actually very simple and already included in Terraform, although a little bit hard to find. You can create a single directory for storing your dynamic providers’ binaries, then set the environment variable “TF_PLUGIN_CACHE_DIR” and you’re done! All Terraform commands run when this variable is set will use a shared directory to lookup and download dynamic providers. That way you can share them across projects and users, if only they have sufficient access permissions to that directory.

Let’s see how it works.

$ rm -rf .terraform
$ export TF_PLUGIN_CACHE_DIR=/opt/terraform-plugin-dir
$ terraform init -upgrade=true
$ ll -h /opt/terraform-plugin-dir/linux_amd64
-rwxr-xr-x 1 piontec piontec 71M May 24 06:28 terraform-provider-aws_v1.20.0_x4
-rwxr-xr-x 1 piontec piontec 12M Apr 19 07:29 terraform-provider-null_v1.0.0_x4
-rwxr-xr-x 1 piontec piontec 12M Apr 19 07:29 terraform-provider-template_v1.0.0_x4
$ ll .terraform/plugins/linux_amd64
-rwxrwxr-x 1 piontec piontec 239 May 24 06:28 lock.json
lrwxrwxrwx 1 piontec piontec 71 May 24 06:28 terraform-provider-aws_v1.20.0_x4 -> /opt/terraform-plugin-dir/linux_amd64/terraform-provider-aws_v1.20.0_x4
lrwxrwxrwx 1 piontec piontec 71 May 24 06:28 terraform-provider-null_v1.0.0_x4 -> /opt/terraform-plugin-dir/linux_amd64/terraform-provider-null_v1.0.0_x4
lrwxrwxrwx 1 piontec piontec 75 May 24 06:28 terraform-provider-template_v1.0.0_x4 -> /opt/terraform-plugin-dir/linux_amd64/terraform-provider-template_v1.0.0_x4
$ du -hs .terraform
336K .terraform

So, after setting the “TF_PLUGIN_CACHE_DIR” variable, Terraform downloads providers to that directory and just links from your project’s “.terraform” directory to the shared directory. And that’s it! Now you can have just a single copy of each provider for the given provider version.

Please keep in mind, that each project still selects the providers’ versions on its own: the fact that one Terraform directory downloaded a new provider version to the cache directory doesn’t mean that all other projects will use the new version as well. So, normal versioning restrictions apply. If you want to be sure which version is locked for use with your current project, you can inspect SHA256 of files saved in one of the files in the “.terraform” directory:

$ cat .terraform/plugins/linux_amd64/lock.json
{
"aws": "3c78df7e116ed60bf917e4cd5cda5999ada163ebffd9142ea41aa7992252cda8",
"null": "d45c0f02cbc08b3915e143434e575298d4c638a2c93598c12f4ea551cd821abd",
"template": "f1d8e373d9f89d21fade8858a562bed75b463c814a2cf8fb750eb017083f1e88"
}

$ ll /opt/terraform-plugin-dir/linux_amd64/
-rwxr-xr-x 1 piontec piontec 72686016 Apr 19 07:29 terraform-provider-aws_v1.15.0_x4
-rwxr-xr-x 1 piontec piontec 73263584 May 7 08:52 terraform-provider-aws_v1.17.0_x4
-rwxr-xr-x 1 piontec piontec 74119744 May 21 14:05 terraform-provider-aws_v1.19.0_x4
-rwxr-xr-x 1 piontec piontec 74242624 May 24 06:28 terraform-provider-aws_v1.20.0_x4
-rwxr-xr-x 1 piontec piontec 11621440 Apr 19 07:29 terraform-provider-null_v1.0.0_x4
-rwxr-xr-x 1 piontec piontec 11711744 Apr 19 07:29 terraform-provider-template_v1.0.0_x4

$ sha256sum /opt/terraform-plugin-dir/linux_amd64/terraform-provider-aws_v1.20.0_x4
3c78df7e116ed60bf917e4cd5cda5999ada163ebffd9142ea41aa7992252cda8 /opt/terraform-plugin-dir/linux_amd64/terraform-provider-aws_v1.20.0_x4

As you can see, the SHA256 hash for AWS provider saved in the “lock.json” file matches the hash of  AWS provider v1.20 saved in the cache directory.

The post Cache your Terraform providers to save space and time appeared first on Tailored Cloud.

• use Terraform’s “null_resource” and optionally “template_file” to easily create config files for other software (like Ansible)
• be careful when you add explicit “depends_on” attribute, as it might result in reporting submodule data as always “changed”, even if they weren’t

1. Introduction

Recently, I had a strange problem with Terraform (all the code below was tested with Terraform 0.11.2). In our company, we use it to manage all our IaaS resources in AWS, but also to generate some configuration files for Ansible. Such configuration files need to be based on data that only Terraform has, so we use “null_resource” to generate them. But in some cases, our “null” type resources were always reported as changed, no matter if the source variables used in the resource changed or not. This quickly became a problem. When you read Terraform plan you really want to know which parts of your infrastructure are going to be changed, including Ansible config files. So, in this blog post, I’m trying to address two things: the former is a simple example of how to generate configuration files for other tools from Terraform. The latter is about nailing down the “always changed null_resource” issue.

2. Working example – generating configuration files

Let’s start to dig into the problem using a simple Terraform code, which allows us to generate an arbitrary configuration file based on Terraform’s configuration and resources. To do that, we combine “template_file” data provider with a “null_resource”  and its “trigger” and “provisioner” properties. In the example below, we have a variable “names”, which contains a comma-separated list of values. For each value, we want to run a template using the values as the template parameter. Next, using “null_resource” and “local-exec” provisioner, we use simple “echo” shell command to output the result of rendering the template into any file we want. That way we can easily generate a whole bunch of config files, although in this example I’m putting all outputs into a single “out.txt” to keep it simpler. Here’s the Terraform code that does just that:

variable "names" {
  default = "joe,jimmy"
}

data "template_file" "init" {
  count = "${length(split(",", var.names))}"
  template = "${file("templates/test.template")}"
  vars {
    test = "${jsonencode(element(split(",", var.names), count.index))}"
  }
}

resource "null_resource" "web" {
  count = "${length(split(",", var.names))}"
  triggers {
    template_rendered = "${join(",", data.template_file.init.*.rendered)}"
  }

  provisioner "local-exec" {
    command = "echo \"${join(",", data.template_file.init.*.rendered)}\" >> out.txt"
  }
}

and the “templates/test.template”:

Hello ${test}

Now we start with typical Terraform bootstrap commands:

$ terraform init
$ terraform apply

which should produce the following output:

data.template_file.init[1]: Refreshing state...
data.template_file.init[0]: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

+ null_resource.web[0]
id: <computed>
triggers.%: "1"
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n"

+ null_resource.web[1]
id: <computed>
triggers.%: "1"
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n"


Plan: 2 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

null_resource.web[1]: Creating...
triggers.%: "" => "1"
triggers.template_rendered: "" => "Hello \"joe\"\n,Hello \"jimmy\"\n"
null_resource.web[0]: Creating...
triggers.%: "" => "1"
triggers.template_rendered: "" => "Hello \"joe\"\n,Hello \"jimmy\"\n"
null_resource.web[1]: Provisioning with 'local-exec'...
null_resource.web[0]: Provisioning with 'local-exec'...
null_resource.web[0] (local-exec): Executing: ["/bin/sh" "-c" "echo \"Hello \"joe\"\n,Hello \"jimmy\"\n\" >> out.txt"]
null_resource.web[1] (local-exec): Executing: ["/bin/sh" "-c" "echo \"Hello \"joe\"\n,Hello \"jimmy\"\n\" >> out.txt"]
null_resource.web[1]: Creation complete after 0s (ID: 4859977991312868035)
null_resource.web[0]: Creation complete after 0s (ID: 2379833441165791621)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

This looks perfectly valid. Let’s check the plan again with “terraform plan”:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.template_file.init[1]: Refreshing state...
data.template_file.init[0]: Refreshing state...
null_resource.web[1]: Refreshing state... (ID: 4859977991312868035)
null_resource.web[0]: Refreshing state... (ID: 2379833441165791621)

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

No changes detected, as can be expected. Perfect. Just to confirm, our “out.txt” file contains now:

Hello joe
,Hello jimmy
Hello joe
,Hello jimmy

3. Refactoring: creating a module

In real life, TerraForm code needs to be organized into modules to keep it more structured and manageable. So let’s start creating a very simple and minimalistic module, that just returns a constant variable for us. We create “modules/dummy/main.tf” file with the following content:

output "file_path" {
  value = "out.txt"
}

We also include this module in our “test.tf” file by adding the following lines:

module "dummy-1" {
  source = "./modules/dummy"
}

Just for testing, let’s remove the configuration file “out.txt” and the state file. Now, let’s run “apply”:

data.template_file.init[1]: Refreshing state...
data.template_file.init[0]: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

+ null_resource.web[0]
id: <computed>
triggers.%: "1"
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n"

+ null_resource.web[1]
id: <computed>
triggers.%: "1"
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n"

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

null_resource.web[0]: Creating...
triggers.%: "" => "1"
triggers.template_rendered: "" => "Hello \"joe\"\n,Hello \"jimmy\"\n"
null_resource.web[1]: Creating...
triggers.%: "" => "1"
triggers.template_rendered: "" => "Hello \"joe\"\n,Hello \"jimmy\"\n"
null_resource.web[1]: Provisioning with 'local-exec'...
null_resource.web[0]: Provisioning with 'local-exec'...
null_resource.web[0] (local-exec): Executing: ["/bin/sh" "-c" "echo \"Hello \"joe\"\n,Hello \"jimmy\"\n\" >> out.txt"]
null_resource.web[1] (local-exec): Executing: ["/bin/sh" "-c" "echo \"Hello \"joe\"\n,Hello \"jimmy\"\n\" >> out.txt"]
null_resource.web[1]: Creation complete after 0s (ID: 6827055886347134598)
null_resource.web[0]: Creation complete after 0s (ID: 7071338054223239370)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

OK, works as expected. Let’s confirm the state by running “plan” again:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.template_file.init[0]: Refreshing state...
data.template_file.init[1]: Refreshing state...
null_resource.web[1]: Refreshing state... (ID: 6827055886347134598)
null_resource.web[0]: Refreshing state... (ID: 7071338054223239370)

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

So far so good.

4. Module dependency and the “always changed” problem

In real life, modules frequently depend on values returned by other modules. Normally, Terraform does a good job at detecting these dependencies automatically. If it misses such dependency and still one module relies on results of the other, you should manually add the dependency information by adding “depends_on” attribute. You might even put it in your code just in case, to be sure the execution order is correct. So, to demonstrate that, let’s add an explicit dependency between our “init” template_file resource and our module. We need to add the line:

depends_on = ["module.dummy-1"]

in our “init” resource section. Let’s ask Terraform what it thinks about this change by running “plan”:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

null_resource.web[1]: Refreshing state... (ID: 6827055886347134598)
null_resource.web[0]: Refreshing state... (ID: 7071338054223239370)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
<= read (data resources)

Terraform will perform the following actions:

<= data.template_file.init[0]
id: <computed>
rendered: <computed>
template: "Hello ${test}\n"
vars.%: "1"
vars.test: "\"joe\""

<= data.template_file.init[1]
id: <computed>
rendered: <computed>
template: "Hello ${test}\n"
vars.%: "1"
vars.test: "\"jimmy\""

-/+ null_resource.web[0] (new resource required)
id: "7071338054223239370" => <computed> (forces new resource)
triggers.%: "1"=> <computed> (forces new resource)
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n" => "" (forces new resource)

-/+ null_resource.web[1] (new resource required)
id: "6827055886347134598" => <computed> (forces new resource)
triggers.%: "1" => <computed> (forces new resource)
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n" => "" (forces new resource)

Plan: 2 to add, 0 to change, 2 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

That’s a bummer. From now on, every “plan” or “apply” run will report both our “init” file resources as changed and they will be regenerated on every Terraform run.

5. Summary

You can definitely use null resource to generate config files for another software. Still, it seems there’s a bug in the current (0.11.2) version of Terraform. It’s present also in few previous versions I checked. So, be careful when you add explicit dependencies between modules and other resources, as this might result in your “null” type resources to be always reported as changed.

The post Generating arbitrary configuration files from Terraform; modules and “always changed” resources. appeared first on Tailored Cloud.

• “selectattr” filter in Ansible is useful for filtering lists based on attributes of the objects in the list

  •  selectattr('name', 'match', 'eth[2-9]') 

• Ansible’s “sum” filter can be used for reducing lists into new objects (including lists)

  • sum(attribute='ips', start=[]) 

Advanced list operations in Ansible

Intro

In the previous entry, I gave you an overview of how you can do all the basic operations on lists in Ansible. Soon after that, I faced a problem, that was still solvable without writing an Ansible plugin. Using list filters was enough, yet the task definitely wasn’t simple. Let me show you what the challenge was and how you can combine some advanced list filters in Ansible to perform more complicated list tasks.

The problem

In our company, we use Ansible to configure some multi-interface multi-IP networking instances running in AWS. These instances are used to manage client’s IPs, so we need to have full control over which IP is assigned where. To deal with that, we have an Ansible role that reads a YAML file in the following format for each of instance’s interfaces:

# file host_vars/HOST/interfaces-eth1.yml
interfaces_eth1:
- gw: 10.10.10.1
  name: eth1
  ips:
  - {ip: 10.10.10.104, owner: TEST, project: The Project}
  - {ip: 10.10.10.105, owner: Our Client, project: The Project} 

 
# file host_vars/HOST/interfaces-eth2.yml 
interfaces_eth2: 
- gw: 10.10.10.1 ips:
  name: eth2
  - {ip: 10.10.10.204, owner: TEST, project: The Project}
  - {ip: 10.10.10.205, owner: Our Client, project: The Project}
  - {ip: 10.10.10.206, owner: Our Client, project: The Project}
  - {ip: 10.10.10.207, owner: Our Client, project: The Project}
  - {ip: 10.10.10.208, owner: Our Client, project: The Project}
  - {ip: 10.10.10.209, owner: Our Client, project: The Project}

Then, we combine all per-interface configs into a single config file for an instance like this:

# file host_vars/HOST/interfaces.yml
interfaces: "{{ interfaces_eth1 }} + {{ interfaces_eth2 }}"

Now, the problem we were facing is: on each instance, we have at least 1 IP address dedicated to monitoring and checking if the given interface is up and running. Such IP address has the owner field set to “TEST”. So, the challenge is to find at least one “TEST” IP on each instance. Basically, we want the first “TEST” IP address out of all addresses assigned to all interfaces of any single networking machine.  Additionally, we have to ignore “eth0” and “eth1” interfaces, as they are being used for other purposes.

The solution, Ansible and Jinja2 way

If you want to test the solution on your local host, create 3 files in the directory ‘host_vars/localhost’ as above. The problem can be solved with the following ansible code. Let’s first have a look and then I will explain it step by step:

# file playbook.yml
- hosts: all
 tasks:
 - set_fact:
 test_ip: "{{ interfaces 
 | selectattr('name', 'match', 'eth[2-9]') 
 | sum(attribute='ips', start=[]) 
 | selectattr('owner', 'equalto', 'TEST') 
 | map(attribute='ip') 
 | list 
 | first 
 | default('NOT_FOUND') }}"
 - debug:
 msg: "The TEST IP is {{ test_ip }}"

In line 5, we start with our list of interface configurations, created from both “interfaces_eth1” and “interfaces_eth2”, which are one-element lists.

Line 6 uses “selectattr” filter, which acts as a filter for a list of objects. This filter passes only objects matching certain criteria. In our case, the two-element list of interface configuration blocks is checked one element at a time and only blocks having attribute “name” matching a regular expression “eth[2-9]” are passed forward. The fact that we want to match by a regular expression is reflected by the 2nd argument set to “match”.

Then, line 7 does a nice trick: it’s a reduction operator that acts on all elements of a list and produces a single resulting object. Its arguments are object’s attribute to act on and the initial state. The operand applied here for reduction is of course “sum”. So, we start with an empty list (the 2nd argument) and then for each interface definition block on our list, we add to the empty list the attribute “ips” of the interface block. The result is a single list of all IP addresses assigned to all interfaces that were not filtered out by the previous line.

Next, in line 8, from all the IPs on the list, we select only ones that have the attribute “owner” equal to “TEST”. Now, things get simpler. We want only the IP address, without “owner” and ” project” attributes, so we “map” every object into a new one by selecting only the attribute “ip”. In line 9 we turn the sequence of IPs into a list again. In line 10 we select just the first element of the list, in case we have found more than one. If we found none, we add “default” filter at the very end.

OK, let’s run it.

$ ansible-playbook -c local -i 'localhost,' playbook.yml

PLAY [all] *********************************************************

TASK [Gathering Facts] *********************************************
ok: [localhost]

TASK [set_fact] ****************************************************
ok: [localhost]

TASK [debug] *******************************************************
ok: [localhost] {
 "msg": "The TEST IP is 10.10.10.204"
}

PLAY RECAP *********************************************************
localhost : ok=3 changed=0 unreachable=0 failed=0 

And it works! Tested with Ansible 2.4.3.0, Jinja2 2.9.6 and python 2.7.14.

The post Advanced list operations in Ansible appeared first on Tailored Cloud.

• • use “select” filter to filter a list and “match” to combine it with reg exps, like:

    "{{ ansible_interfaces | select('match', '^(eth|wlan)[0-9]+') | list }}"

  • use “map” to map list elements, like:

    "{{ ansible_interfaces | map('upper') | list }}"

  • use just “+” operator to combine two lists into one, like:

    "{{ ansible_interfaces + [\"VETH-1\", \"VETH-2\"] }}"

• • check here for more advanced example: Advanced list operations in Ansible

Ansible filters and lists operators

Ansible provides a rich set of filters, which you can apply to your variables. Recently, I needed to filter and map a list of host interfaces. I couldn’t remind myself how to do this, so I jumped to ansible filters docs. But I couldn’t find it there. Well, you have to remember, that expression evaluation in Ansible is based on Jinja2, so you need to check Jinja2 filters documentation as well. You will find basic operators there.

When operating on lists, functional languages are a great example: by combining just a few list operators, you can get almost any desired transformation of the list. These basic operators include:

• filter – to select just matching elements from the list and create a new list from them
• map – to convert all elements on the list according to transformation function that operates on a single element
• reduce – to convert (aggregate) all elements of a list into a single value
• flat map – to merge multiple lists into a single list

Let’s now see how to perform them in Ansible 2.4.

Our example – list all network interfaces

As our example list we want to transform in ansible, let’s use a list of local network interfaces that Ansible discovers during setup and stores in “ansible_interfaces” list. A simple play that just fetches and prints the list can look like this:

- hosts: all
  tasks:
    - name: print interfaces
      debug:
        msg: "{{ ansible_interfaces }}"

Let’s save the above code in file ‘list_operators.yml’ and run it with:

ansible-playbook -c local -i 'localhost,' list_operators.yml

On my PC this produces the following output:

PLAY [all] ****************************************************************

TASK [Gathering Facts] ****************************************************
ok: [localhost]

TASK [print interfaces] ***************************************************
ok: [localhost] {
    "msg": [
        "docker0", 
        "lo", 
        "veth3e8c318", 
        "wlan0",
        "eth0", 
        "vethe2459c9"
    ]
}

PLAY RECAP ****************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

Filter a list in Ansible

Let’s assume we want to filter the interfaces list to include only “eth” or “wlan” types of interfaces. As you can find here in Jinja2 docs, we can use the “select” filter to filter the list. But the great thing – and not that easy to find – is that we can combine “select” with “match” operator to test for matching against a regular expression! This brings us to the following solution:

- hosts: all
  tasks:
    - name: print interfaces
      debug:
        msg: "{{ ansible_interfaces | select('match', '^(eth|wlan)[0-9]+') | list }}"

So, what it does is that it passes each element on the list “ansible_interfaces” to a “match” test. This test checks with the regexp if the element starts with “eth” or “wlan” followed by at least 1 digit. Elements that match are combined back again into a list with the “list” filter. The code above produces the following output:

TASK [print interfaces] ***************************************************
ok: [localhost] {
    "msg": [
        "wlan0", 
        "eth0"
    ]
}

Map list elements in Ansible

OK, let’s now see how we can use the map filter in Ansible. Let’s assume that we want to convert each interface name on our list to upper case only. Jinja2 has a filter for that, so our Ansible task becomes:

- hosts: all
  tasks:
    - name: print interfaces
      debug:
        msg: "{{ ansible_interfaces | map('upper') | list }}"

and produces the following output:

TASK [print interfaces] ***************************************************
ok: [localhost] {
    "msg": [
        "DOCKER0", 
        "LO", 
        "VETH3E8C318", 
        "WLAN0", 
        "DOCKERCON", 
        "ETH0", 
        "VETHE2459C9"
    ]
}

Merge two lists into one in Ansible

Now, what if we want to merge two lists into a single one that contains elements of both of them? That’s really easy, as the normal “+” operator works for lists and does exactly that. So, running this code:

- hosts: all
  tasks:
    - name: print interfaces
      debug:
        msg: "{{ ansible_interfaces + [\"VETH-1\", \"VETH-2\"] }}"

produces the following output, where the list of local interfaces is extended with elements “VETH-1” and “VETH-2” from the second list:

TASK [print interfaces] ***************************************************
ok: [localhost] {
    "msg": [
        "docker0", 
        "lo", 
        "veth3e8c318", 
        "wlan0", 
        "dockercon", 
        "eth0", 
        "vethe2459c9", 
        "VETH-1", 
        "VETH-2"
    ]
}

Aggregate / reduce

Here the problem is trickier – it depends on what you really want to do with list’s elements. Probably the most popular aggregation filter is “join”, which just joins list elements into a single stream with elements separated by a given separator. There are a few more, but in general, when you need something more complicated, you either have to escape to a full Jinja2 template and do something custom in a “{% for %}” loop or implement your own filter.

Related

This article also has a continuation: Advanced list operations in Ansible

The post How to filter, join and map lists in Ansible appeared first on Tailored Cloud.