• Set “TF_PLUGIN_CACHE_DIR”  environment variable to an empty dir, then rerun “terraform init” to switch to a shared providers directory. This allows to cache Terraform providers.

Dynamic Terraform providers

Have you noticed, that when you run “terraform init”, Terraform fetches additional binaries from the Internet? These binaries are called “dynamic providers” and they are (mainly) interfaces to the infrastructure operators you’re using in your Terraform code.

So, for example, when you’re using AWS as an infrastructure provider, and you also use some templating to generate config files, you might run “terraform version” and see something like this:

$ terraform version
Terraform v0.11.7
+ provider.aws v1.20.0
+ provider.null v1.0.0
+ provider.template v1.0.0

In the example above, we have 3 providers being used: the “null” and “template” provider and a specialized provider for talking with AWS API. You can find a list of projects developing these providers on Github. But where does Terraform keep these additional provider binaries? Well, each directory that is used as a root of Terraform project (where you run “terraform init”) has a hidden “.terraform” directory. There you can find the downloaded providers:

ll -h .terraform/plugins/linux_amd64
-rwxrwxr-x 1 piontec piontec 239 May 24 06:28 lock.json
-rwxr-xr-x 1 piontec piontec 71M May 24 06:28 terraform-provider-aws_v1.20.0_x4
-rwxr-xr-x 1 piontec piontec 12M Apr 19 07:29 terraform-provider-null_v1.0.0_x4
-rwxr-xr-x 1 piontec piontec 12M Apr 19 07:29 terraform-provider-template_v1.0.0_x4

This approach is nice, as all your projects are independent and self-contained: you can copy the whole directory to some other place or even machine and everything will still work. However, this approach has also a drawback. If you have multiple Terraform working directories, you have to download and store your providers for every one of them. As you can see, the basic set of providers listed above takes around 100MB – not much for today standards, but still. This gets even worse when you have multiple Terraform projects used by a group of people.  And when every one of them runs “terraform init”, he or she needs to wait for the binaries to be downloaded.

A cache directory for providers to the rescue

The solution to the above problem is actually very simple and already included in Terraform, although a little bit hard to find. You can create a single directory for storing your dynamic providers’ binaries, then set the environment variable “TF_PLUGIN_CACHE_DIR” and you’re done! All Terraform commands run when this variable is set will use a shared directory to lookup and download dynamic providers. That way you can share them across projects and users, if only they have sufficient access permissions to that directory.

Let’s see how it works.

$ rm -rf .terraform
$ export TF_PLUGIN_CACHE_DIR=/opt/terraform-plugin-dir
$ terraform init -upgrade=true
$ ll -h /opt/terraform-plugin-dir/linux_amd64
-rwxr-xr-x 1 piontec piontec 71M May 24 06:28 terraform-provider-aws_v1.20.0_x4
-rwxr-xr-x 1 piontec piontec 12M Apr 19 07:29 terraform-provider-null_v1.0.0_x4
-rwxr-xr-x 1 piontec piontec 12M Apr 19 07:29 terraform-provider-template_v1.0.0_x4
$ ll .terraform/plugins/linux_amd64
-rwxrwxr-x 1 piontec piontec 239 May 24 06:28 lock.json
lrwxrwxrwx 1 piontec piontec 71 May 24 06:28 terraform-provider-aws_v1.20.0_x4 -> /opt/terraform-plugin-dir/linux_amd64/terraform-provider-aws_v1.20.0_x4
lrwxrwxrwx 1 piontec piontec 71 May 24 06:28 terraform-provider-null_v1.0.0_x4 -> /opt/terraform-plugin-dir/linux_amd64/terraform-provider-null_v1.0.0_x4
lrwxrwxrwx 1 piontec piontec 75 May 24 06:28 terraform-provider-template_v1.0.0_x4 -> /opt/terraform-plugin-dir/linux_amd64/terraform-provider-template_v1.0.0_x4
$ du -hs .terraform
336K .terraform

So, after setting the “TF_PLUGIN_CACHE_DIR” variable, Terraform downloads providers to that directory and just links from your project’s “.terraform” directory to the shared directory. And that’s it! Now you can have just a single copy of each provider for the given provider version.

Please keep in mind, that each project still selects the providers’ versions on its own: the fact that one Terraform directory downloaded a new provider version to the cache directory doesn’t mean that all other projects will use the new version as well. So, normal versioning restrictions apply. If you want to be sure which version is locked for use with your current project, you can inspect SHA256 of files saved in one of the files in the “.terraform” directory:

$ cat .terraform/plugins/linux_amd64/lock.json
{
"aws": "3c78df7e116ed60bf917e4cd5cda5999ada163ebffd9142ea41aa7992252cda8",
"null": "d45c0f02cbc08b3915e143434e575298d4c638a2c93598c12f4ea551cd821abd",
"template": "f1d8e373d9f89d21fade8858a562bed75b463c814a2cf8fb750eb017083f1e88"
}

$ ll /opt/terraform-plugin-dir/linux_amd64/
-rwxr-xr-x 1 piontec piontec 72686016 Apr 19 07:29 terraform-provider-aws_v1.15.0_x4
-rwxr-xr-x 1 piontec piontec 73263584 May 7 08:52 terraform-provider-aws_v1.17.0_x4
-rwxr-xr-x 1 piontec piontec 74119744 May 21 14:05 terraform-provider-aws_v1.19.0_x4
-rwxr-xr-x 1 piontec piontec 74242624 May 24 06:28 terraform-provider-aws_v1.20.0_x4
-rwxr-xr-x 1 piontec piontec 11621440 Apr 19 07:29 terraform-provider-null_v1.0.0_x4
-rwxr-xr-x 1 piontec piontec 11711744 Apr 19 07:29 terraform-provider-template_v1.0.0_x4

$ sha256sum /opt/terraform-plugin-dir/linux_amd64/terraform-provider-aws_v1.20.0_x4
3c78df7e116ed60bf917e4cd5cda5999ada163ebffd9142ea41aa7992252cda8 /opt/terraform-plugin-dir/linux_amd64/terraform-provider-aws_v1.20.0_x4

As you can see, the SHA256 hash for AWS provider saved in the “lock.json” file matches the hash of  AWS provider v1.20 saved in the cache directory.

The post Cache your Terraform providers to save space and time appeared first on Tailored Cloud.

• use Terraform’s “null_resource” and optionally “template_file” to easily create config files for other software (like Ansible)
• be careful when you add explicit “depends_on” attribute, as it might result in reporting submodule data as always “changed”, even if they weren’t

1. Introduction

Recently, I had a strange problem with Terraform (all the code below was tested with Terraform 0.11.2). In our company, we use it to manage all our IaaS resources in AWS, but also to generate some configuration files for Ansible. Such configuration files need to be based on data that only Terraform has, so we use “null_resource” to generate them. But in some cases, our “null” type resources were always reported as changed, no matter if the source variables used in the resource changed or not. This quickly became a problem. When you read Terraform plan you really want to know which parts of your infrastructure are going to be changed, including Ansible config files. So, in this blog post, I’m trying to address two things: the former is a simple example of how to generate configuration files for other tools from Terraform. The latter is about nailing down the “always changed null_resource” issue.

2. Working example – generating configuration files

Let’s start to dig into the problem using a simple Terraform code, which allows us to generate an arbitrary configuration file based on Terraform’s configuration and resources. To do that, we combine “template_file” data provider with a “null_resource”  and its “trigger” and “provisioner” properties. In the example below, we have a variable “names”, which contains a comma-separated list of values. For each value, we want to run a template using the values as the template parameter. Next, using “null_resource” and “local-exec” provisioner, we use simple “echo” shell command to output the result of rendering the template into any file we want. That way we can easily generate a whole bunch of config files, although in this example I’m putting all outputs into a single “out.txt” to keep it simpler. Here’s the Terraform code that does just that:

variable "names" {
  default = "joe,jimmy"
}

data "template_file" "init" {
  count = "${length(split(",", var.names))}"
  template = "${file("templates/test.template")}"
  vars {
    test = "${jsonencode(element(split(",", var.names), count.index))}"
  }
}

resource "null_resource" "web" {
  count = "${length(split(",", var.names))}"
  triggers {
    template_rendered = "${join(",", data.template_file.init.*.rendered)}"
  }

  provisioner "local-exec" {
    command = "echo \"${join(",", data.template_file.init.*.rendered)}\" >> out.txt"
  }
}

and the “templates/test.template”:

Hello ${test}

Now we start with typical Terraform bootstrap commands:

$ terraform init
$ terraform apply

which should produce the following output:

data.template_file.init[1]: Refreshing state...
data.template_file.init[0]: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

+ null_resource.web[0]
id: <computed>
triggers.%: "1"
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n"

+ null_resource.web[1]
id: <computed>
triggers.%: "1"
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n"


Plan: 2 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

null_resource.web[1]: Creating...
triggers.%: "" => "1"
triggers.template_rendered: "" => "Hello \"joe\"\n,Hello \"jimmy\"\n"
null_resource.web[0]: Creating...
triggers.%: "" => "1"
triggers.template_rendered: "" => "Hello \"joe\"\n,Hello \"jimmy\"\n"
null_resource.web[1]: Provisioning with 'local-exec'...
null_resource.web[0]: Provisioning with 'local-exec'...
null_resource.web[0] (local-exec): Executing: ["/bin/sh" "-c" "echo \"Hello \"joe\"\n,Hello \"jimmy\"\n\" >> out.txt"]
null_resource.web[1] (local-exec): Executing: ["/bin/sh" "-c" "echo \"Hello \"joe\"\n,Hello \"jimmy\"\n\" >> out.txt"]
null_resource.web[1]: Creation complete after 0s (ID: 4859977991312868035)
null_resource.web[0]: Creation complete after 0s (ID: 2379833441165791621)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

This looks perfectly valid. Let’s check the plan again with “terraform plan”:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.template_file.init[1]: Refreshing state...
data.template_file.init[0]: Refreshing state...
null_resource.web[1]: Refreshing state... (ID: 4859977991312868035)
null_resource.web[0]: Refreshing state... (ID: 2379833441165791621)

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

No changes detected, as can be expected. Perfect. Just to confirm, our “out.txt” file contains now:

Hello joe
,Hello jimmy
Hello joe
,Hello jimmy

3. Refactoring: creating a module

In real life, TerraForm code needs to be organized into modules to keep it more structured and manageable. So let’s start creating a very simple and minimalistic module, that just returns a constant variable for us. We create “modules/dummy/main.tf” file with the following content:

output "file_path" {
  value = "out.txt"
}

We also include this module in our “test.tf” file by adding the following lines:

module "dummy-1" {
  source = "./modules/dummy"
}

Just for testing, let’s remove the configuration file “out.txt” and the state file. Now, let’s run “apply”:

data.template_file.init[1]: Refreshing state...
data.template_file.init[0]: Refreshing state...

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

+ null_resource.web[0]
id: <computed>
triggers.%: "1"
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n"

+ null_resource.web[1]
id: <computed>
triggers.%: "1"
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n"

Plan: 2 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

null_resource.web[0]: Creating...
triggers.%: "" => "1"
triggers.template_rendered: "" => "Hello \"joe\"\n,Hello \"jimmy\"\n"
null_resource.web[1]: Creating...
triggers.%: "" => "1"
triggers.template_rendered: "" => "Hello \"joe\"\n,Hello \"jimmy\"\n"
null_resource.web[1]: Provisioning with 'local-exec'...
null_resource.web[0]: Provisioning with 'local-exec'...
null_resource.web[0] (local-exec): Executing: ["/bin/sh" "-c" "echo \"Hello \"joe\"\n,Hello \"jimmy\"\n\" >> out.txt"]
null_resource.web[1] (local-exec): Executing: ["/bin/sh" "-c" "echo \"Hello \"joe\"\n,Hello \"jimmy\"\n\" >> out.txt"]
null_resource.web[1]: Creation complete after 0s (ID: 6827055886347134598)
null_resource.web[0]: Creation complete after 0s (ID: 7071338054223239370)

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

OK, works as expected. Let’s confirm the state by running “plan” again:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.template_file.init[0]: Refreshing state...
data.template_file.init[1]: Refreshing state...
null_resource.web[1]: Refreshing state... (ID: 6827055886347134598)
null_resource.web[0]: Refreshing state... (ID: 7071338054223239370)

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

So far so good.

4. Module dependency and the “always changed” problem

In real life, modules frequently depend on values returned by other modules. Normally, Terraform does a good job at detecting these dependencies automatically. If it misses such dependency and still one module relies on results of the other, you should manually add the dependency information by adding “depends_on” attribute. You might even put it in your code just in case, to be sure the execution order is correct. So, to demonstrate that, let’s add an explicit dependency between our “init” template_file resource and our module. We need to add the line:

depends_on = ["module.dummy-1"]

in our “init” resource section. Let’s ask Terraform what it thinks about this change by running “plan”:

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

null_resource.web[1]: Refreshing state... (ID: 6827055886347134598)
null_resource.web[0]: Refreshing state... (ID: 7071338054223239370)

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
<= read (data resources)

Terraform will perform the following actions:

<= data.template_file.init[0]
id: <computed>
rendered: <computed>
template: "Hello ${test}\n"
vars.%: "1"
vars.test: "\"joe\""

<= data.template_file.init[1]
id: <computed>
rendered: <computed>
template: "Hello ${test}\n"
vars.%: "1"
vars.test: "\"jimmy\""

-/+ null_resource.web[0] (new resource required)
id: "7071338054223239370" => <computed> (forces new resource)
triggers.%: "1"=> <computed> (forces new resource)
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n" => "" (forces new resource)

-/+ null_resource.web[1] (new resource required)
id: "6827055886347134598" => <computed> (forces new resource)
triggers.%: "1" => <computed> (forces new resource)
triggers.template_rendered: "Hello \"joe\"\n,Hello \"jimmy\"\n" => "" (forces new resource)

Plan: 2 to add, 0 to change, 2 to destroy.

------------------------------------------------------------------------

Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.

That’s a bummer. From now on, every “plan” or “apply” run will report both our “init” file resources as changed and they will be regenerated on every Terraform run.

5. Summary

You can definitely use null resource to generate config files for another software. Still, it seems there’s a bug in the current (0.11.2) version of Terraform. It’s present also in few previous versions I checked. So, be careful when you add explicit dependencies between modules and other resources, as this might result in your “null” type resources to be always reported as changed.

The post Generating arbitrary configuration files from Terraform; modules and “always changed” resources. appeared first on Tailored Cloud.