When company standards fight with the tools.

Who wins?

This is a more recent story for once, but it's something that should be shared now.

What happens when you standardize a company's way of setting up their Jenkins Pipelines, Terraform code, and Ansible playbooks? Great things, and reproducibility, no?

Absolutely.

What happens when your standardization is done in a vacuum? Possibly good things still, but more chances for issues down the line.

What happens if you develop those standards without accounting for the standards of the tools the company is using? Mistakes.

The company I currently work for has a pretty decent team of engineers who help shape code and tool standards. There's a YAML file that every team needs to use if they want to leverage the Jenkins Library and a few certification processes. It's kinda designed to act as an SBOM for deployments. It's a rather elegant solution (albeit annoying when you're not used to using it) to a problem that things like well-designed Helm charts are meant to solve, or ignition files for CoreOS.

packages:
  docker:
    images: {}
  rpms: {}
  vms:
    image: some-image-reference-string-like-ec2
...

Above is a rough example of what it looks like. Straightforward, to the point. Naming convention could be better. The idea was “how can deployments be certified, while standardizing how teams interact with the data?”, and it simply acts as a reference for all the deployment tools. Terraform cares about the vm images. Jenkins can use a predefined container for tasks. Ansible can also use the containers if it's a docker host, as well as pulling a list of what rpm packages are needed to set up the host.

The issue probably still isn't clear. It's just a YAML file, Terraform can reference it directly for vars, so can Jenkins and Ansible, right? Yes but the question becomes this … how do you reference it for all the tools without duplicating it?

Terraform and Jenkins don't care, just give relative paths and it's golden. Ansible, on the other hand … has a dozen ways you could reference the file, all ranging from good to bad. What's the best way to standardize it across the company in the simplest way? Symlinks.

If this was live, I'd ask for a show of hands …

“How many of you are familiar with all the ways to generate facts about a system using Ansible?”

“Now, how many are familiar with the way to generate facts for specific things like packages and services?”

How many hands would I lose on the second? Majority, probably. Yes, there is in fact 3 modules that will generate facts for a host with data that's not normally gathered as part of gather_facts or setup module. They are mount_facts, package_facts, and service_facts. There's also 2 Windows-specific facts, but I'm less familiar with running Ansible against Windows targets. They each will add to host_vars with following keys respectively: mounts, packages, and services.

So, if you've symlinked a YAML file, whose parent key is also packages, and a role or a task in your playbook also calls ansible.builtin.package_facts, which values would you see if you were to do a debug on the var packages?

The module package_facts would win because it's updating host facts. This has to do with the hierarchy of variable precedence. Anything symlinked into group_vars will be replaced by anything added as a host fact, or handled as include_vars.

So, what's the best way to prevent the collision?

You have a few options. You could do include_vars in your playbook, but the safest option is to add something like the following to your group_vars (preferably under all since this is a mostly static file)

manifest: "{{ lookup('file', 'path/to/package_manifest.yml') }}"

Now, you could use a relative path to the file, but my preference is to do absolute, or using "{{ playbook_dir }}" as the relative starting point. For example, if your project layout is the following:

ansible/
|—— inventory/
|   |—— <envs>
|       |—— group_vars
|—— roles/
|   |—— myrole/
|       |── tasks/
|       |── templates/
|
|—— files
|—— playbooks/
│   |—— deploy.yml
package_manifest.yml

Your lookup path would be playbook_dir + '/../package_manifest.yml'.