This is a more recent story for once, but it's something that should be shared now.

What happens when you standardize a company's way of setting up their Jenkins Pipelines, Terraform code, and Ansible playbooks? Great things, and reproducibility, no?

Absolutely.

What happens when your standardization is done in a vacuum? Possibly good things still, but more chances for issues down the line.

What happens if you develop those standards without accounting for the standards of the tools the company is using? Mistakes.

The company I currently work for has a pretty decent team of engineers who help shape code and tool standards. There's a YAML file that every team needs to use if they want to leverage the Jenkins Library and a few certification processes. It's kinda designed to act as an SBOM for deployments. It's a rather elegant solution (albeit annoying when you're not used to using it) to a problem that things like well-designed Helm charts are meant to solve, or ignition files for CoreOS.

packages:
  docker:
    images: {}
  rpms: {}
  vms:
    image: some-image-reference-string-like-ec2
...

Above is a rough example of what it looks like. Straightforward, to the point. Naming convention could be better. The idea was “how can deployments be certified, while standardizing how teams interact with the data?”, and it simply acts as a reference for all the deployment tools. Terraform cares about the vm images. Jenkins can use a predefined container for tasks. Ansible can also use the containers if it's a docker host, as well as pulling a list of what rpm packages are needed to set up the host.

The issue probably still isn't clear. It's just a YAML file, Terraform can reference it directly for vars, so can Jenkins and Ansible, right? Yes but the question becomes this … how do you reference it for all the tools without duplicating it?

Terraform and Jenkins don't care, just give relative paths and it's golden. Ansible, on the other hand … has a dozen ways you could reference the file, all ranging from good to bad. What's the best way to standardize it across the company in the simplest way? Symlinks.

If this was live, I'd ask for a show of hands …

“How many of you are familiar with all the ways to generate facts about a system using Ansible?”

“Now, how many are familiar with the way to generate facts for specific things like packages and services?”

How many hands would I lose on the second? Majority, probably. Yes, there is in fact 3 modules that will generate facts for a host with data that's not normally gathered as part of gather_facts or setup module. They are mount_facts, package_facts, and service_facts. There's also 2 Windows-specific facts, but I'm less familiar with running Ansible against Windows targets. They each will add to host_vars with following keys respectively: mounts, packages, and services.

So, if you've symlinked a YAML file, whose parent key is also packages, and a role or a task in your playbook also calls ansible.builtin.package_facts, which values would you see if you were to do a debug on the var packages?

The module package_facts would win because it's updating host facts. This has to do with the hierarchy of variable precedence. Anything symlinked into group_vars will be replaced by anything added as a host fact, or handled as include_vars.

So, what's the best way to prevent the collision?

You have a few options. You could do include_vars in your playbook, but the safest option is to add something like the following to your group_vars (preferably under all since this is a mostly static file)

manifest: "{{ lookup('file', 'path/to/package_manifest.yml') }}"

Now, you could use a relative path to the file, but my preference is to do absolute, or using "{{ playbook_dir }}" as the relative starting point. For example, if your project layout is the following:

ansible/
|—— inventory/
|   |—— <envs>
|       |—— group_vars
|—— roles/
|   |—— myrole/
|       |── tasks/
|       |── templates/
|
|—— files
|—— playbooks/
│   |—— deploy.yml
package_manifest.yml

Your lookup path would be playbook_dir + '/../package_manifest.yml'.

There's nothing special I think in how I'm running things, or doing my prep, but I think it's a good example of how someone new getting started could do things. I'm going to make a couple assumptions, like having a basic screen, but mostly break it up into shared tools, on-line, and off-line play.

Shared Tools

These are kind of the ones that I use regardless if I'm running the session in person or on-line.

D&D Beyond

I unfortunately was introduced to playing using dndbeyond, and have mostly bought into using it on a regular basis for managing digital books and the campaigns. Mostly for the books and to share what I've purchased with the players. I have lots of gripes with 'Wizards of the Coast' with how they've integrated the new 2024 edition into it. Other than not giving me an option for restricting which content I get to use (there's no saying no to the 2024 SRD), I like how simplifies managing the character if you're new. As the DM the campaign screen is invaluable to have quick access to everyone's AC and current HP.

Obsidian

Obsidian has a lot of great features, with a pretty great and robust plugin system. I can definitely say I'm not leveraging nearly as much as I could.
Since it's really just allowing me to write in Markdown format, with some extra features added, I don't have to do much other than write what I want and organize whatever is making sense for me.

I prefer to group single encounters, setting it up so that the scene is one document, creatures/combat is another page, and miscellaneous stuff that might be useful. I also have a separate folder for any world building information I'm working on, mostly broken up into generic plots for everyone, character specific stuff, as well as tracking background stuff that is either to happen or will be happening.

There's a couple plugins that I could not get by without, mostly the dice roller and fantasy statblocks.

<cover why dice roller is great, mostly the formulaic sidebar and random tables with a roller built-in>

Fantasy Statblocks is by far a required item, if you are reskinning an existing or even using it for something that exists already. By default, it only has access to creatures from the SRD, but has the ability to import JSON objects into it's bestiary from a variety of different sources. I went and added all the creatures I have access to from the books I own using a site that is a little gray. Each statblock I add to my notes is editable by modifying fields.

Above is an example of a standard Green Hag. Let's tweak it slightly … 100 HP, less AC just because and definitely needs to be named Greenie McWitchface of the famous Waterdeep McWitchfaces.

```statblock
creature: Green Hag
name: Greenie McWitchface
dice: false
hp: 100
ac: 12
```

Dice set to false is really only needed because I have it set to randomly roll based on the hit dice. So, if I want it to be hard-coded to 100, I have to tell it not to roll.

You hear the rustle of a bush a ways away but chalk it up to the wind. As you finish lighting the fire for the evenings meal, 3 dire wolves leap out of the bush, with one landing a bite attack on the druid, pinning them to the ground. Roll for initiative...

It's a small on-the-fly setup for an encounter, but you get the idea. But you kinda figured something like that was coming based on the title. Predictability in my blog posts is kinda expected at this point.

That's not the start of the story. For that we are going to have to take a trip back to April/May 2023. One of my friends mentioned they had friends moving back to the area and was looking to get a D&D game started and wanted to know if interested. Me, never having played and only some minor interest or exposure, said sure. A few weeks later I was creating my first D&D character. I created Shammagar, a Kobold barbarian whose way too strong and angry for his own good. We played most of the summer before schedules got to crazy, lending to protracted breaks between sessions.

I was enthralled. 80% of my TV time was bingeing Critical Role S3. Come fall, I found a campaign that was starting up at a local game store / bar.

That's it.

Pretty boring right? Expected something more?

That's just how I got into it. Becoming a DM is kind of a sillier story filled with dumb mistakes.

We find ourselves in the middle of Oct, I'm just thinking about what I want to do for my annual Charity stream with Extra Life. I had a couple friends participating that were D&D-curious and decided I wanted to try and run a one-shot for them. So I did the sane thing, helped them create a couple of characters, picked a pre-written adventure and ran it during the event. What a sane thing to do right?

WRONG!

I may or may not have drunk the Kool-Aid a bit too strongly. I decided I was going to try and write a one-shot for them completely out of the blue, with custom maps and everything.

The maps honestly turned out fantastic, I spent dozens of hours decorating them. The story was simple, had a clear villain and a clear resolution with a few paths of variety to make things either easier or harder. From there, that's when things started falling apart. In my head, the whole thing should have taken 3–4 hours. By hour 5 they were just getting to the last encounter.

I wrote too much, bit off more than I could chew. Honestly I kinda felt defeated for a long while. I didn't try to pick up a virtual DM screen again for a long time.

Fast-forward to March 2024. Spring break is coming up, our tables' DMs are going to be gone for about 2 weekends. I offer to run so that anyone at the table who wants to still play while they're gone can.
I had been paying more attention to how much would get done in the 2-ish hour window we'd been playing in, and had gotten my hands on One-Shot Wonders from Roll & Play Press. I had them pick an adventure based on a d100 roll, and started translating the prep into my laptop to make life easier. I erred more on easier for the encounter design, and they finished the whole thing in about 90 minutes. The next session I picked at random based on their level and left it on the defaults, finished in a little over 2 hours. Everything went back to normal for a bit until finals, and they ended the campaign in a glorious battle.

Meanwhile ... on a Discord server I'm a part of, a bunch of people had interest in starting up a game, and I was sort of volunteered to run the campaign. I didn't mind really, busy lives meant it would be a once-a-month type thing. I decided I was going to run Keys from the Golden Vault as a base for them. That campaign has been going good, but slow.

I was starting to feel a little more comfortable telling a story and fitting it into a certain timeframe. I didn't want to run one-shots all summer and I had splurged a few months back on something ... daunting.

It's a fantastic, modified system put together by Free League Publishing, based on The One Ring 2e. Key word being modified. A lot of the base is the same, probably 80% of it. Changes to certain checks, different rules for travel, palaver, and Shadow Points make enough tweaks that it's not as simple as "create some new characters, and we'll drop them into the world and everything will run smoothly".

Except with it being a low magic setting, eldritch blasts and spells in general would really just make most encounters go too smoothly. So I made a few adjustments to bridge the rules to basic 5e rules.

• Limit the class options to Bard, Rogue, Ranger, with the options of Paladin and Cleric coming from the Callings in the book.
• Limited the subclasses and the spells available, re-flavoring them to be more skills than conjuring healing out of thin air, for example.
• Kept some of the traveling rules, but cut the Fellowship and Council phases, treating them more like average downtime and NPC encounters
  • For keeping the difficulty of getting stuff out of Council type meetings, I decided to keep the intentions and still have them make basic persuasion or intimidation rolls based on expected DC's for what kind of support they could expect.
• Kept most of the checks the same as regular 5e, re-merging the 3 skills split from Nature, but keeping Medicine as Intelligence (one of the LotR changes).
• Cut out the idea of Shadow Points

Honestly it was hard and stressful. but SO rewarding. About a month or so into it, I started seeing a shift in my storytelling abilities, I started loosening up a bit more about the rules and going with the flow. I eventually ended up including the Shadow Points back into the story after one of the players decided to go full chaotic. Dropping a ceiling on your party members and refusing to pull them out of the rubble deserves some consequences.

Which brings me to the present and the last few months. I'm back to running one-shots for them in person. The Fall/Spring campaign has been on hold, and I've been too busy with life stuff to focus on the conversions again. Did create a murder mystery event that's run all of October for them, but that's the closest to a campaign they've gotten to do in the last couple of months.

I did start another campaign though. Two of my friends who did the stream one-shot wanted to play again, then an old coworker expressed interest. 2 of my friend's friends wanted to play. And then one of their relatives. Quickly, within 6 weeks (3 sessions), I had a table of 6 playing virtually. That campaign is a whirlwind, with the understanding from them that I'm using them to tweak and adjust more of my storytelling skills. I decided I wanted to do more work to tie in backstories into the campaign and make actual plot points around it. So I find myself every other week building a narrative that's going to spawn months.

I'm starting to feel outside my depths, and probably biting off way too much, but I'm realizing so far the only pressure to perform is on myself. As things happen in the campaign I might write about them, I don't want to spoil much, but I will say that in 25 days in-game the campaign will no longer be really following Candlekeep Mysteries.

I've got half finished blogs for ditching docker compose, migrating to Podman, Podman networking differences and solutions for a k8s-style deployment of Podman all lined up and in various states of completion. Unfortunately … I've got no interest in completing them right now.

I think my plan for a bit is to change gears for a while. Instead of tech projects, or intricacies of SystemD, I'm going to try to write about a few other passions I have. There's probably going to be A LOT of D&D talk, occasionally books/TV/Movies/Video Games, and the rarer occasion a look into some self-care hobbies. Don't worry, the hobbies are mostly PG, rare occasion of PG-13 for when the shirts come off 😏.

Not going to make a habit of this, but shout-out to my friend and old co-worker Neel. He pointed me at a TheVerge article about Ghost (the platform used for this blog) starting up the process for ActivityPub support. I'm a large fan of the Fediverse moment and design and will be definitely supporting it once it's fully baked.

Anyways, where was I? Right … Neel. I mentioned that I hadn't really touched this in a year, and he was kinda like “you should do it again”. He also gave the nudge I needed to think about rebranding all this and not quite starting from scratch, but shifting what this would be about. Without him, this would undoubtedly probably have stayed dead for a long while.

Life's had it's ups and downs, inside and out of work. Eventually I might share some of it.

Consider this a placeholder for some rambling to come soon.

A few months back when I started writing up content, the blog platform software that I use to run the blog, Ghost, started not subtly letting me know that the latest version of the software was available. With a catch, it was not compatible with my current database. When I spun up the blog originally, MariaDB was the more performant option, with little to no differences between the original MySQL. MariaDB has been making great strides in continuing to improve performance. Although, this means the feature-set is going to continue to shift. I saw something a few months back, regarding the next version of MySQL was going to be fundamentally different enough, that MariaDB would start to no longer be the drop-in replacement. (I didn't save the link because it didn't seem relevant at the time ☹️).

(drake meme, with ghost logo, MariaDB vs MySQL)

Ghost seems to be relying on certain features of MySQL for newer features that don't exist in MariaDB. Which means, if I want to keep my platform updated and secure, I will have to put in the work to do this migration. Migrating and upgrading also means a chance to change some underlying things, and redesign workflows.

Current stack vs New stack

Now, fundamentally, there's no major changes.

The current stack is as follows:

Docker via docker-compose

• Traefik (webserver)
• mariadb (db)
• ghost (website)
• watchtower (updater)
• Matomo (monitoring and analytics)

Now, I liked the stack when I originally used it, but for other projects Traefik felt more like a slog. Matomo never worked properly the way I wanted it to. Everything else has been stable.

The new stack:

Docker via Ansible

• Caddy (webserver)
• mysql (db for ghost)
• ghost (website)
• watchtower (container updater)
• shynet (analytics)
• postgres (db for shynet)
• telegraf/grafana (monitoring)
• giscus (comments)

There are a few new faces involved. Some of it I'll go into deep detail with other posts, like using Ansible instead of compose, as well as some other new tricks I picked up through this process.

Monitoring via telegraf and grafana is still in flux, I'll eventually hone down what I want exactly, and will cover it at a later date.

The second-biggest difference, while the smallest in the amount of work, is the switch to Caddy from Traefik. Don't get me wrong, Traefik works wonderfully, but the config was finicky in my experience. It's truly designed with a docker or Kubernetes focus first, which made it hard for me to implement it back when I was running a mix of LXC instances and containers outside the VPS the blog runs on. At the time, there also wasn't a clean way to do DNS challenges for the certs. I think the config has grown a bit and some new features have been added to improve QoL.

I chose Caddy over Nginx with certbot, mostly because I want the cert renewal and everything more integrated. Personally, I prefer using nginx as an ingress controller or in a more traditional environment setup. Besides, the Caddyfile makes it really easy to spin up sites with reverse proxies:

blog.jonnagel.us {
  encode gzip
  
  proxy_pass $container_name:2368
}

That's it! Just add more blocks as needed for different domains. DNS auth for LetsEncrypt works slightly differently, but not enough to really cover right now. Their documentation is top-notch in my opinion. I have a script that I use for my other VPS's that I'll share in a different post how to somewhat automate the DNS creation with caddy installed and managed by systemd. It could easily be modified to work with a docker environment.

Now, the most exotic thing on that list is probably giscus. Disqus back in 2017 made it only possible to remove scripts and ads from their system for a fee of $10/month originally, now it's $11/month. While that's might seem pretty affordable, I don't have nearly enough traffic per month to justify it, or get a cut from the ad revenue. Ads and injections scripts are known to carry malware, and I personally do not feel comfortable to potentially expose others to that kind of risk.

That's where giscus comes in. It's an interesting app that allows you to leverage GitHub Discussions as a comment system. That means, you don't need to build and manage your own database for users, deal with spam or bots. Authentication is managed by GitHub, making it simple to use. There's a bit of configuration to do to get it setup, but adding it onto the template for the posts was quite simple.

This is all that's needed:

<script src="https://giscus.app/client.js"
            data-repo="org/reponame"
            data-repo-id="XXXXXXXXXXXX"
            data-category="Q&A"
            data-category-id="XXXXXXXXXXX"
            data-mapping="pathname"
            data-strict="0"
            data-reactions-enabled="1"
            data-emit-metadata="0"
            data-input-position="top"
            data-theme="preferred_color_scheme"
            data-lang="en"
            crossorigin="anonymous"
            async>
        </script>

This will get generated by the app's site, and will for the most part walk you through what's needed to get it setup. I just added some additional CSS configuration to get it mostly set up the way I wanted it. I might to more configuration later, so I don't need as many modifications to get it to look how I want. For now, it looks good enough for my needs.

Now, for the meat of the migration. Moving from MariaDB to MySQL. Now, normally, such a migration should be a piece of cake in theory, but there were 2 major things I needed to take care of as part of the migration.

1. Collation schema updates
2. Manage the additional bloat or memory that MySQL uses compared to MariaDB.

The first one was an unfortunate issue. Like I mentioned, the steps to migrate are quite simple.

1. Stop Ghost
2. Take a mysqldump backup.
3. Spin down MariaDB container and delete volume.
4. Create new MySQL container and volume, with backup script mounted as part of the initdb.d folder.
5. Wait a little bit and start Ghost again.

Unfortunately, due to the collation changes, I was prompted with errors like the following:

MigrationScriptError: alter table `posts_products` add constraint `posts_products_post_id_foreign` foreign key (`post_id`) references `posts` (`id`) on delete CASCADE - UNKNOWN_CODE_PLEASE_REPORT: Referencing column 'post_id' and referenced column 'id' in foreign key constraint 'posts_products_post_id_foreign' are incompatible.

It took some googling, and I was able to parse that the error. MariaDB 10 normally uses the default schema of latin1_swedish_ci, but since the original database setup was caused by an initialization script as part of Ghost, the collation was set to utf8mb4_general_ci when I did my initial installation. MySQL 8 changed their default uft8 collation to utf8mb4_0900_ai_ci.

So, what is one to do? Do a fresh install, migrate the posts one by one and update the dates somehow? Do I update the script from the SQL dump to have the correct collation value? I prefer to work smarter than harder 🙃, so I went with the update the script method, similar to how Andrej did theirs. Instead of using sed, I leveraged the global replace feature in vi. Afterwards, starting up the database and consequently Ghost went smoothly after. Or so I thought…

For reasons, I have been running the blog on a simple $5/month DigitalOcean droplet for a few years now. It's all been running fine for 4 years. During the process of upgrading the OS to the latest LTS and deploying the stack, everything started to hang. Looking at htop in a different terminal, I was able to see that load was low, but memory was completely filled. Since there's not normally swap space on droplets, OOM Killer was killing things temporarily (hooray for restart: unless stopped 🤦). Took a look at docker stats, MySQL was using nearly 550 MB of memory! That will not do, so my options are to either increase the size of the droplet, or try to perform some memory tuning. This time, working smarter is memory tuning, and probably stupidly perform some memory limits on the container. I had docker set the memory limit to 512M.

It's probably not the most efficient config, but here's what I added to the my.cnf that gets applied:

connect_timeout         = 5
wait_timeout            = 600
max_allowed_packet      = 16M
thread_cache_size       = 0
sort_buffer_size        = 32K
bulk_insert_buffer_size = 0
tmp_table_size          = 1K
max_heap_table_size     = 16K

myisam_recover_options = BACKUP
key_buffer_size         = 2M
table_open_cache        = 400
myisam_sort_buffer_size = 512M
concurrent_insert       = 2
read_buffer_size        = 16K
read_rnd_buffer_size    = 16K

slow_query_log_file     = /var/log/mysql/mariadb-slow.log
long_query_time = 10

binlog_expire_logs_seconds        = 259200
max_binlog_size         = 100M

default_storage_engine  = InnoDB
# you can't just change log file size, requires special procedure

#innodb_log_file_size   = 50M
innodb_buffer_pool_size = 64M
innodb_log_buffer_size  = 2M
innodb_file_per_table   = 1
innodb_open_files       = 400
innodb_io_capacity      = 400
innodb_flush_method     = O_DIRECT

[client]
socket              = /var/run/mysqld/mysqld.sock

[mysqldump]
quick
quote-names
max_allowed_packet      = 16M

I'll probably tweak it the longer it runs, but for now it seems to tame the beast enough. Memory appears to max out for the container around 240 MB and when everything is idle it settles down around 70 MB.

The total memory usage now is around 750 MB at it's peak.

The only other new thing in the stack is an application called Shynet. It's a very modern, privacy analytics tool. It doesn't use cookies, and respects the DNT setting in browsers. From the GitHub page, here are the exact metrics that it will collect:

* Hits — how many pages on your site were opened/viewed
* Sessions — how many times your site was visited (essentially a collection of hits)
* Page load time — how long the pages on your site look to load
* Bounce rate — the percentage of visitors who left after just one page
* Duration — how long visitors stayed on the site
* Referrers — the links visitors followed to get to your site
* Locations — the relative popularity of all the pages on your site
* Operating system — your visitors' OS (from user agent)
* Browser — your visitors' browser (from user agent)
* Geographic location & network — general location of your visitors (from IP)
* Device type — whether your visitors are using a desktop, tablet, or phone (from user agent)

Looking at how it's configured to work, it uses a combination of an invisible pixel to get the basic information about your session (IP for geolocation, browser type, and OS/Device Type). Then it uses a simple script to track load times and everything else. Now, I personally have it set so that IP is not tracked if you don't have DNT set. I don't need that information, knowing where you might be from and checking out my blog is more than enough.

It took a while to get to this point, mostly because there's no clear documentation to do all the things that I wanted to piece together.

Let's take a recap of what I'm trying to do; A year ago I got some new hardware to build a new server to replace my existing one, and a couple of months ago I had a few hiccups in finishing the build-out. This server is to replace my slow and extremely long in the tooth HP Microserver, and to act as a local server and learning playground.

Next up has been figuring out what was needed to automate the creation of the VMs for the Kubernetes cluster (as well as any other VMs I plan on deploying). Now, there's a fairly well documented set of tools that I already used previously for setting up the libvirt bits. I already had gone and set up the storage pools and the networking to allow DHCP on my LAN. Unfortunately, that's where the documentation starts to fall apart. To do the installation of a VM, you use the community.libvirt.virt module.

Here, you do the following:

1. Have a disk already created, and know the path to it.
2. Any additional resources you need created (network, shared disk, etc.)
3. Have an XML domain definition file you can feed to the playbook

That's all the information that's given. 1 and 2 are pretty obvious, it's a similar situation to deploying a cloud VM with something like terraform. Unfortunately, this means, if you don't know how to make a minimal domain definition file. This gets even more obfuscated once you try to also do this while using Fedora CoreOS as the guest OS.

That changes today!

No seriously, I'm going to walk you through how to make a bare-bones definition file, what you need for it to have Fedora CoreOS as the OS, and a bare-bones ignition file to build your VM.

Libvirt's Domain Definition

I unfortunately was not able to find any real “here's a real basic XML file you can use” or “here are the bits you require in the file for it to truly work properly” for the VM definition. There is the whole fat domain specification, but that's too much for starting out. I did find a page that was “here run this role that contains this file”, but that's not good enough for me. So, let's take a look at what I used, and break down what I can.

Let's work our way down.

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>{{item.node_name}}</name>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://fedoraproject.org/coreos/stable"/>
    </libosinfo:libosinfo>
  </metadata>
...
</domain>

<domain> is the top-level tag everything will live under. <name> is going to be the name of the domain (VM). It's how you would reference with the virsh command. <metadata> tag is a little less clear, the docs point to it being used depending on what application is creating the domain.

<memory unit='MiB'>{{item.vmem}}</memory>
<vcpu placement='static'>{{item.vcpus}}</vcpu>

<memory> is straightforward. It's the allocation for the memory being made available to the VM. unit can be changed between KiB, MiB and GiB (maybe more, but I don't have enough ram on the host to test that 😉) depending on your needs. I still prefer to do it with MiB for the most part. <vcpus> is also straightforward, in practice. If you want the bare-bones usage, the way I have it with placement set to static and then define the number of cores to be available. If you're looking to manage NUMA layouts, core pass through, I strongly recommend checking this section of the specification.

<cpu mode='host-passthrough' check='none' migratable='on'/>
<os>
  <type arch='x86_64' machine='pc-q35-6.2'>hvm</type>
  <boot dev='hd'/>
</os>

<cpu> is more for when you want to emulate different versions of CPU from what you're running. For what I'm using, host-passthrough is the basic mode, it says “send all the features available to the host CPU through”. This is the section that you would use to define if you wanted to emulate an ARM CPU (or an x86 CPU while on ARM). The <os> tag will be also where you would define most of these specifications for more customization for the host OS. I didn't dive deep into using this, I figured out that I needed arch set to 'x86_64', machine set to 'pc-q35-6.2' and then the inner tag set to hvm. Mostly due to that's how VMs created with the cockpit-machine package sets it, along with virt-install, on my system.

<features>
  <acpi/>
  <apic/>
  <vmport state='off'/>
</features>
<clock offset='utc'>
  <timer name='rtc' tickpolicy='catchup'/>
  <timer name='pit' tickpolicy='delay'/>
  <timer name='hpet' present='no'/>
</clock>

<features> is for the hypervisor to know what features are available. <acpi> allows for better power management, while <apic> allows for interrupts to be sent. Other useful tag would be <pae> for larger memory addresses (useful for 8gb ram on a 32bit OS), <hyperv> if you're dealing with Windows guests.

<clock> is to mimic passing through the internal clock of the motherboard to the VM, allowing you to define your own timezone offset based on the time on your system.

<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
  <suspend-to-mem enabled='no'/>
  <suspend-to-disk enabled='no'/>
</pm>

<on_poweroff>, <on_poweron>, <on_crash> and <pm> tags define how the host OS should handle managing the guest for power states, as well as if the guest should have the ability to sleep.

Devices

Everything under <devices>is everything I decided is needed at a minimum. Some like memballoon are required for KVM/QEMU guests.

<disk> is required for every disk you want to add to your guest. With each disk you need to give the location, whether it be a block device, ISO, raw file, or a qcow2 image file and an address. Even though I originally wanted to do LVM devices, I chose to switch to qcow2 files for now.  Until I can figure out how to do automation better for LVM devices (nearly all the tools are designed around only qcow2 or IMG files right now, with LVM or other block devices being considered advanced features).

<controller> shouldn't be needed since we're doing basic device design, allowing libvirt to pick properly, but I left it since there was no reason to remove it. <channel> is used for host-guest communication. It's not explicitly required, but it makes it easier for graceful shutdown and sharing devices or directories with the guest.

Network <interface>, like disk, is defined for each network interface you want added to your guest OS. These are relatively straightforward as well, and the documentation has some great examples of different use cases. I'm using libvirt networks, hence why the source is using a network type. If you wanted to, you could predefine the mac address the interface would use, but if it's not defined, libvirt will generate it automatically.

<console> is required if you want to be able to access the vm using the command virsh console vm-name-here, I chose to keep it as PTY instead of switching to TTY or virtio-serial. <input> with mouse and keyboard will allow you to interface with vm over console. The <video> tag more than likely is not needed, I haven't played around not including it. It's definitely required if you want to be able to connect over VNC using spice graphics, which is overkill for a headless vm. <rng> is also not needed, but it makes random number generation quicker by using pass-through.

Now, for the pièce de résistance, the magic sauce to get Fedora CoreOS (or RedHat CoreOS, or Flatcar Linux) working. Defining <qemu:commandline> and then adding the argument, -fw_cfg that are passed along to define the kernel parameters to inject the ignition file by the second line with name=opt/com.coreos/config,file=/path/to/ignition/file. Where /path/to/igniton/file is a path to an ignition file on your system.

Ansible Jinja Templating and disk creation.

So, if you noticed, in a few places I have things like {{name}} or {{vcpus}} inside the example above. This is due to making this config into a jinja template, so I can use the same template for all my cluster VMs. To make life easier, inside my vars file I added something that looks like this:

k8s_node:
  - node_name: master1
    vcpus: 2
    vmem: 4096
    disk:
      pool: bulk
      size: 20
    networks:
      - external
      - default
    is_master: true

In the playbook I have it looping through k8s_node.

is_master is currently not being used. Right now, it's acting more like a mental placeholder for me once I get to actually setting up the cluster install.

For networks, I loop through in the template to create the block for both external and default.

I'm not currently doing more than one disk, but I'd handle it similarly to networks after a bit of massaging.

As for the creation of the boot disk, this took a little time to figure out properly. The VM installation of Fedora CoreOS requires the use of the actual vm image, or at least a copy of it. Following this section from the Fedora Docs, while looking up how virt-install works, I realized that backing-store part of the command was creating a copy of the disk and resizing it to the supplied value.

To mimic this behavior, I added 2 separate tasks into the playbook. The first makes a copy of the base image I fetch, the second resizes it to match the value I have inside the vars for that VM.

The Ignition File

An ignition file is what you use to define the setup and configuration of the OS. It's similar to how cloud-init works. In there you can define users, advanced network config (like static IPs), repos and disk configurations.

To get this working as part of the VM, it's a 2 part process. The first, a butane file, and generating the ignition file.

Butane

Butane is a specific YAML-based config file. You can write the files without anything special, but to validate and convert into ignition file you will need the application. There are various ways to install it, but my preferred way is to run it via podman/docker.

The simplest Butane config you can create is one that defines just the user and an ssh key.

variant: fcos
version: 1.4.0
passwd:
  users:
    - name: somename
      ssh_authorized_keys:
        - ssh-rsa AAAA...

I'm taking things a step further, I want everything that's required for running something like kubespray in place. That means I need to do the definition for CRI-O, the Kubernetes repo and a few other things. I didn't figure this out on my own, that credit goes to Carmine.

variant: fcos
version: 1.4.0
passwd:
  users:
    - name: nagelxz
      ssh_authorized_keys:
        - ssh-rsa ...
      home_dir: /home/nagelxz
      password_hash: ...
      groups:
        - wheel
      shell: /bin/bash
kernel_arguments:
  should_exist:
    # Order is significant, so group both arguments into the same list entry.
    - console=tty0 console=ttyS0,115200n8
storage:
  files:
    # CRI-O DNF module
    - path: /etc/dnf/modules.d/cri-o.module
      mode: 0644
      overwrite: true
      contents:
        inline: |
          [cri-o]
          name=cri-o
          stream=1.17
          profiles=
          state=enabled
    # YUM repository for kubeadm, kubelet and kubectl
    - path: /etc/yum.repos.d/kubernetes.repo
      mode: 0644
      overwrite: true
      contents:
        inline: |
          [kubernetes]
          name=Kubernetes
          baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
          enabled=1
          gpgcheck=1
          repo_gpgcheck=1
          gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
            https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    # configuring automatic loading of br_netfilter on startup
    - path: /etc/modules-load.d/br_netfilter.conf
      mode: 0644
      overwrite: true
      contents:
        inline: br_netfilter
    # setting kernel parameters required by kubelet
    - path: /etc/sysctl.d/kubernetes.conf
      mode: 0644
      overwrite: true
      contents:
        inline: |
          net.bridge.bridge-nf-call-iptables=1
          net.ipv4.ip_forward=1
    - path: /etc/hostname
      overwrite: true
      contents:
        inline: master-potato-fries

After you convert that butane file into an ignition file and run the playbook, it'll take about 2–5 minutes to generate the VMs and be available on the IPs that are created. It's an additional setting I need to do, to have Ansible generate the inventory file for the cluster. But I'm leaving that for the tweaks that come in as part of building the next part for running kubespray as part of the playbook.

Lets start with the good. I've automated 80%, got it stood up and have migrated most of my data over to it. All the important, everyday bits are being run from it instead of my old microserver.

The bad, well it's more like tepid, stale water. Lets go in order of things that went wrong or skipped.

The Bad

Kickstart

Yeah, the core of building the server got axed. Attempts were made, but for some reason I could not get the server to recognize the boot option, or when I did, it complained about issues that I did not have with testing on VMs. Instead of dwelling on this one, I've chosen to skip indefinitely. It was more of a nice to have than a need. the disk layout and config options aren't complex enough to have to worry about messing it up if I have to redo from scratch for some reason.

Snapshots

Yeah... Now we're crossing into breaking core designs, but the good news is, this is just skipped "for now". This is partially from originally wanting to do the base of the system as openSUSE and switching back to Fedora. openSUSE makes it incredibly easy to setup and use snapper out of the box. On Fedora, it's easy with a giant asterisk. It's the biggest pain in the world to automate on Fedora 36. Whatever you're thinking of to be the biggest pain, double it. There is a fairly straighforward guide to follow but I have not had luck automating it the way it needs to be done to get it to work.

The issue comes from the need to make very specific kernel modifications because of the filesystem defaults in Fedora. First, SUSE_BTRFS_SNAPSHOT_BOOTING=true needs to be added to the grub config, the boot kernel needs to be rebuilt to support this change. Next you need to change the subvolume ids for the /.snapshot volume so that you can control how the system boots. If these steps are in any way wrong (including trying to skip rebooting to set part of it), you lose access to the system.

I'm not skipping it, eventually I will get this part done, even if it's unfortnately done manually. Mainly, this is going to be needed when I start doing system upgrades, doing more ad-hoc work with containers, and backing up the VMs in the cluster.

Plex Harware Encoding

This one is confusing. Plex is being run inside a Podman container. Nearest I can tell, the official plex container doesn't play nicely with Podman. I mostly has to do with how podman does the hardware passthrough compared to Docker. Docker does passthrough of the device, Podman utilizes hooks into the device to share the abilities. For some reason when the device is successfully passed through, running nvidia-smi returns command not found. Nor does Plex see that there's the option for hardware encoding.

I need more time to investigate this, mostly at a time when Plex can be down and not interrupting anything (which sucks because that's how i've been watching most of my content lately).

Updates? Kinda

I'll be honest, I didn't plan exactly how I would be running updates on things. In order this is what needs to be done

1. Update the core system (security or all)
2. Update containers (currently Plex and Jellyfin)
3. VM upgrades (separation for cluster VMs and adhoc ones, but this is something I haven't spent any time figuring out since I haven't built the cluster part yet).

A week or so ago, I did make a general playbook to do the updates to the system, along with tags to differentiate between security updates and full updates that will have a reboot. That went mostly fine, but it did show me that I had an issue with the containers and the systemd files that were created. I haven't spent time figuring out how I would fix them, yet, but it's on my list of things to do before the end of the year.

The VM part, I have thoughts about how I'm going to update the cluster, but for other vms, I'm either going to use the same playbook for the core system, or I'm going to build in running updates via console somehow (I doubt it, I don't wand to deal with expect if I can avoid it).

The Cluster itself

This one is more of a cop-out than anything 😉. I have the files needed to make the xml domain definitions, but i haven't had the time to tweak them for what I need and then build out the variables. I should also be able to leverage some info from some playbooks at work if I get stuck. Again, it's more of a scope of time, than an issue getting it working.

It's slow going, mostly due to the time I spend working on it, but the system is building out mostly as planned.

So far I've been playing with using Fedora Workstation as the base and trying to leverage podman for containerization and playing with QEMU as the virtualization platform. So far it's been great, incredibly performant and exactly what I'm looking f

First up is being able to reinstall the system in within an hour and have it configured and ready to go. I decided to take a page out of work's playbook. Automate the installer with a kickstart file, finish off with salt or ansible for anything that can't be handled in the %post step.

Kickstart is a configuration file created by RedHat around the time of RedHat Linux 6.2 (possibly older, but this is the fist reference I can find.) to help automate the installer. Back then, the adding of the kickstart file would create what was called a Kick-Me disk. The setup has gone through a few revisions since then, and most distributions support it, but the concept is the same - include a kickstart file, either on disk, another piece of removable media, or on a webserver. It's most notably the kickstart configuration is used as part of PXE boot environments.

In the kickstart file, you can define everything from disk partitioning and network setup to a what packages you want installed by default on first boot. There's also steps to happen either pre or post install. You can find all the option references in this guide.

The kickstart file I'm going to leverage will be a modified one that was generated by the anaconda installer  from my inital setup of the server. To test the configuration, I've been creating VMs in QEMU and modifying the iso I want to use as I go. There's been 2 major revisions so far. The first was a basic config that also tried to install snapper and configure it, the second worked on setting up the partitions exactly without any post s

Above is the test kickstart file I was using ... Lets break it down.

url --mirrorlist="https://mirrors.fedoraproject.org/metalink?repo=fedora-34&arch=x86_64"

# Use graphical install
graphical

# Keyboard layouts
keyboard --xlayouts='us'
# System language
lang en_US.UTF-8
# System timezone
timezone America/New_York --utc

# Run the Setup Agent on first boot
firstboot --enable

Instead of pulling from the iso, I chose to pull from the the default mirror for fedora. So I could watch the process in case it got stuck, which happened a lot in the beginning, I left it in graphical mode. Next bit is just your normal language and timezone settings. Finally is the firstboot. This allows you to go through the inital setup after boot (think in KDE environments, you configure your inital account after the install is complete). I'll be able to remove this in my final product, since the user will be created differently.

# Generated using Blivet version 3.3.3
ignoredisk --only-use=vda
# Partition clearing information
clearpart --all --initlabel
# Disk partitioning information

part btrfs.1 --fstype="btrfs" --ondisk=vda --size=15360
part btrfs.2 --fstype="btrfs" --ondisk=vda --size=59389
part /boot --fstype="ext4" --ondisk=vda --size=2048 --label=boot
part biosboot --fstype="biosboot" --ondisk=vda --size=2
btrfs /vol_root --label=root_vol btrfs.1
btrfs /home_pool --label=fs_pool btrfs.2
btrfs / --subvol --name=@root LABEL=root_vol
btrfs /home --subvol --name=@home LABEL=fs_pool

Now we're getting to the disk setup part. We only want the installer to worry about vda, the disk that I created to install the VM onto. If I'm using a disk over, I'll want to wipe it by using the clearpart command. This will wipe the disk and reinitalize it for use.

Now we're into the juice of the partitioning. I created the VM disk to be 75GB, to test giving a small root partition and a separate home partition on different btrfs volumes for testing snapper rollbacks. Being a vm, i have to setup the biosboot partition. There's also an issue with setting up /boot on a btrfs volume and interacting with grub currently, especially when snapshots are involved. I decided to expose the core btrfs volumes to make the snapper setup easier later.

#Root password
rootpw --lock

user --name=nagel --password=XXXXXXXXXXXXX --groups=wheel

Simplest section. login to root user is locked. My user is defined. There's an additional setting to add the sshkey to login to the account that's created that I'm probably going to add.

%packages
@^server-product-environment
@headless-management
@container-management
@server-hardware-support
@virtualization
ansible
openssh-server
vim
git
python3-dnf-plugin-snapper
snapper
%end

Next is the packages section. This allows me to define both what fedora install i want (mostly due to the nature of using url step instead of cdrom), as well as the packages I want included from start. Everything that starts with an @ symbol is a group of packages to install. headless-management adds packages to support cockpit, container-management adds all the packages needed for podman, server-hardware-support adds things like lm_sensors and other utils to monitor the health of the server, virtualization installs qemu and supporting packages, and the rest of the packages are things I want as part of the setup of my install. This list will change as I finalize the core install.

There's also the option for pre and post install scripts. %pre is usesful if you have specific partion schemes you want setup that are not easily done with the default partitioning commands. %post is there for anything you want in place after the installation is finished. I tried to use it for the snapper config, but even running outside of the chroot, I could not get it to work.

I'll keep tweaking the kickstart file as I go, and start building the ansible or salt configuration to finalize the setup before moving to the vm setups and other items.

A couple days ago, I was listening to Coder Radio episode 448, and they mentioned the whole faker.js and colors.js fiasco that went down last week. Work's kept me quite busy this past week or so, so I missed the whole thing as it was unfolding.

Short recap:

The developer and creator of 2 popularly used JavaScript libraries updated both in drastically different ways. faker.js he removed the data and replaced it with a readme that says "What really happened with Aaron Swartz?", with colors.js the version number was bumped to 6.6.6 and entered an infinite loop printing garbage data into the console after 3 lines of "LIBERTY LIBERTY LIBERTY". NPM reverted the version and hit github account was suspended (access was restored later). News articles from the likes of forbes, and other people in online circles resorted to calling him a terrorist for these actions.

These actions that he performed on his own code do not make him a terrorist.

FULL STOP.

He's done some things in his past that can classify him as a potential terrorist (if you want to learn what, that's on you to figure out. This is a story of a larger problem, not a man's need for specialized help), but it does not make it right to label his current actions as such.

faker.js and colors.js were packages made my this person, he technically is in his right to do whatever he wants with the code up on GitHub and push to NPM.

The issue, as I see it, is companies like Amazon have used his projects both internally and included as part of AWS development kits for prototyping things to run on lambda. The issues roll down to the creator and starts to feel burned out on top of other issues going on in his life.

I'm with him on this. If a project like this is so integral to how they, or their services function, there should be compensation. I don't think 6-figure contract is fair, but definitely more than a couple thousand thrown their way (if at all).

The same thing could be said about all the development efforts of log4j. With the 0-day vulnerability that showed up back in December, thousands of companies and applications were made vulnerable. Even the main developer for that project had gotten little in terms of donations prior to the 0-day. Prior to the 0-day being announced, there were no sponsors, that slowly changed after the release of the CVE on the 10th of december. Now, they have 74 individual sponsers, but that doesn't help secure the future of an integral piece of code that thousands of companies are using in their applications. In the case of Log4j, there are about 6 companies supporting his efforts on the sponsor page, but that's not nearly enough. The team I was formerly on supported various different development teams that were using Log4j in the project and while I can't be sure, it's a good chance that nobody at the company, or the company itself has made a donation. While Ralph Goers is employed as a Software Architect, that should not diminish the fact that it takes time and effort to write software.

faker.js creator shares a similar story. There's only 10-12 consistent sponsors prior to the incident on the 7th of January. There's 48 now, but none of the public sponsors are corporations.

What should you, as an individual do?

If you can afford to, you should donate to any Opensource developers or projects that are integral to your everyday workflow. Any large video game streamer should seriously consider donating to the OBS Project on the regular. If you can't afford to, go out of your way to try and support the developer or project in anyway you can. It could be as simple as a thank you, helping fix a bug, or helping them improve documentation. Start being the change in the cycle.

What should coporations be doing?

Donating a lot more than they do now. Hiring integral contributors and maintainers of projects that their company relies on. Some companies already do this. Facebook employs one of the developers of BTRFS. wolfSSL employs the sole founder and lead developer of cURL. Amazon contributes back to the Rust language. Those are the ones I know off the top of my head.

As an individual inside large companies, you should try and push for a change on what kind of donations they make. It won't be easy to change the norm.

The name Free and Open Source Software is a blessing and a curse at the same time. This model has greatly allowed for people like me to excel in their professional career, allowing us to learn on and off the job and not being locked into a specific company's way of doing things. But it's also the bane of anyone who's written a popular piece of software, even if it's not integral to a large corporation. Many open-source developers end up experiencing some form of burnout when the projects get too large and users are asking of too much from them.

Us as users should try and be more respectful of these individuals time and effort that they put into these projects.

On and off for the last 5 years I've had a personal project (eventually I'll go into detail) I pick up for a bit when i decide it's time to give it another shot to completion.

The first rendition was done with a traditional LAMP stack, with codeigniter, twig and a few other things. It took me about 4 months to get the project to 70%, putting in a couple hours here and there. A vast majority of that time was spent learning a lot of the stack to the point I could do something functional with it. Unfortunately, I didn't have the bandwidth to  bring it to completion, as I'm allergic to JavaScript and to do what I wanted was outside of my skill level at the time.

Fast-Forward to 2019. I've started a new job, I'm living on my own, and I find just enough free time to try again but differently. This time I choose the stack to be Python framwork Falcon with a Postgres database, mostly based off of performance benchmarks i've seen (all benchmarks are lies, don't listen to them), while finally embracing that Docker lifestyle and made it probably 40% of the way to completion. I had inserting and fetching based off of api endpoints working, but little else. This time I decided to only focus on backend work instead of trying to make a partially working frontend as well. This time it was shelved due to personal things keeping me from wanting to explore more of what I was writing.

That brings us to current day. I've been spending a lot of my work time dealing with APIs and their design and decided I should practice what I preach and start over. Starting over was mostly because trying to pick up the Falcon framework again after working in Bottle was a pain. And the performance of Bottle had mostly caught up to Falcon, to the point it's not a big difference. I kept the same backend library I had been writing, so everything was mostly plug and play and got back to the same 40% in less than a day. And that's when the fire nation attacked........

Just kidding. I spent whatever free time and energy I had improving on the backend code, reaching what I would consider feature parody to other similar projects I know of. Part of this work includes a type of expiration table, to be driven by a stored procedure to routinely delete data based on values in the main table. This was always in the design of the project and probably considered crucial in the end product. I hadn't done the stored procedure as part of the LAMP stack because i was focused on getting the frontend where I wanted it instead of that functionality, but I was familiar of how to do what I wanted, so it was a non-issue.

This is when I started to hit into my issues. Stored procedure to delete items? Easy Peasy. Option to schedule stored procedure? 404...

Wait what? Yes, by default, Postgres doesn't contain the option to schedule a stored procedure internally. Missing this when I decided to pick Postgres is on me, but in defense, I never dreamed that I'd have to include "can schedule stored procedures" in my look for other RDBMS options outside of MySQL. I had worked with MSSQL and assumed anything in this day and age would allow this (besides sqlite, but that's expected based on design since it's not a client-server database).

I spent the time doing the research for my options if I wanted to stick with Postgres:

• try to bake a cron to run inside the postgres container
• create a function inside my backend code to run on a cronschedule on a different process thread to trigger the stored procedure
• ditch the concept of scheduling, make the application delete the rows on fetch if the citeria is met (slow and expensive)
• install the pg_cron extension as part of the postgres image and bake the schedule as part of the inital sql that runs when spunt up with docker-compose

I decided the most portable option was number 3. If I chose to switch RDBMS platforms it would require the least amount of work to migrate. Honestly it went great until I tried to activate the schedule.

2021-07-25 22:29:49.825 UTC [84] ERROR:  can only create extension in database postgres
2021-07-25 22:29:49.825 UTC [84] DETAIL:  Jobs must be scheduled from the database configured in cron.database_name, since the pg_cron background worker reads job descriptions from this database.
2021-07-25 22:29:49.825 UTC [84] HINT:  Add cron.database_name = 'database_name' in postgresql.conf to use the current database.
2021-07-25 22:29:49.825 UTC [84] CONTEXT:  PL/pgSQL function inline_code_block line 4 at RAISE
2021-07-25 22:29:49.825 UTC [84] STATEMENT:  CREATE EXTENSION pg_cron;
psql:/docker-entrypoint-initdb.d/init.sql:3: ERROR:  can only create extension in database postgres
DETAIL:  Jobs must be scheduled from the database configured in cron.database_name, since the pg_cron background worker reads job descriptions from this database.
HINT:  Add cron.database_name = 'database_name' in postgresql.conf to use the current database.

Well ..... Shit.

So for most installs of postgrest, that doesnt seem to be that big of a deal. Just use the default database as your database in a container, and off to the races, right? Well it used to be bad practice to build your application in the default database, hence me defaulting to using database_name. What about having your inital script create database_name, switch to it and then setup the tables and whatnot?

Postgres doesn't have or understand the USE command. everything talks about while connecting with psql to do \c database_name and then run the script, or imbed the switch and script you want to run inside another script.

No thank you... Not after spending 2 hours on getting both a working stored procedure and going down the rabbit hole to find this out.

I'll say again, these limitations are things I should have discovered much earlier into the process, and it's my fault for not doing the footwork until most of my design was set in stone.

For this project I'll be giving up Postgres, but only because I don't want to jump through these hoops to keep using it for this project. I'm sure I'll have other things I'll work on where it makes perfect sense. So, for now, goodbye Postgres ... it was kinda fun working with you.

Whether you're working on a traditional REST API or building something out using AWS Gateway, eventually (read: you should be doing it from the start) you may need to start documenting your functions and endpoints.

At work we've been building out API for standardizing various automation activities, and building actions for our internal chatbot. The setup the REST API has mostly been done from a few members of our team  as a lift and shift from various modules inside our NodeJS / Hubot framework. Some of the functions are simple, comparing versions of applications with healthcheck endpoints in environments, to stuff that's more complex, redeploying all applications in an environment.

As of right now, there's no standardization on the request or response structures, nor are there any example requests for new team members to pick up working on making changes to the functions, just examples on how to use it from the chat platform. The best solution on how to fix both missing examples and standardize (at most) the requests is to create an REST API Specification document. While there's a few different REST API specifications out there, the most common is Swagger / OpenAPI. This specification can be easily designed and tested in something like Swagger UI, Insomnia, or other REST design tools.

What's required to build out an specification document?  A basic one is quite simple to get setup. Lets take a look:

openapi: 3.0.3
info:
  title: just a simple api spec
  version: 0.0.1
  description: can be whatever you want, usually i expand on the title
  contact: 
    email: some.email@example.com
servers:
  - url: some.url.example.com
    description: if you are using something like swagger, this will give you a place to test against.
paths: 
  /say_hi
    get:
      description: returns hello
      responses:
        '200':
          content:
            application/text:
              schema:
                type: string
                example: "hello"

In this simple example you can see that there is a single endpoint path described, /say_hi that when you hit the url along with the path, (some.url.example.com/say_hi) you could expect to get back "hello". That's really all that's happening here.

Lets take it a step further, and say you have a function behind our API that could be hit to get the status of a bunch of healthchecks at once from your servers in the NYC datacenter. The response you'd expect would be a json object list where it lists the host name and if it's up or down as follows:

{
	{"host_1" : "UP"},
	{"host_2" : "UP"},
	{"host_3" : "DOWN"}
}

Lets see what the path description would be:

paths:
  /healthchecks/{datacenter}:
    decscription: return the healthcheck status for the provided datacenter.
    parameters:
      - in: path
        name: datacenter
          schema:
            type: string
        required: true
    responses:
      '200':
        description: successful call for datacenter healthcheck
        content:
          application/json:
            schema:
              type: object
              properties:
                 hostname:
                   type: string
                   example: "UP"
      '404':
        description: datacenter was not found
        content:
          application/json:
            schema:
              type: object
              properties:
                datacenter:
                  type: string
                  example: "San_Diego"
                message: 
                  type: string
                  example: "Could not find datacenter"

It takes some time to plan out what your response objects may be, so don't be afraid to change your spec as you go along. In this case we've also defined what the object might look like if you want a different responce for when the datacenter is not found.

There's a few things you need to keep in mind as your are building your API and designing he responses:

• Try to work on the path documentation as your going along. It doesn't need to be finished right from the start. If you know the post object is going to be a json object, but don't know how many fields it's going to have, you can have it commented out and build it as you go along.

• Your input and output may change! Don't feel stuck because what you thought your input or output was going to look like isn't what ended up being what you invisioned. Let your specification grow with your api

  • At work our API was originally designed to drive the complex functions for the chatbot, and thus, the response objects were originally designed to be easier to inject inside the chatbot framework. As we decided it was easier to leverage the API for some jenkins jobs instead of duplicating functionality we've had to adapt and are now serving responses based off of header information.

• Descriptions of what the path is for is immensely helpful. Someone who is looking at your swagger page, or the actual document itself should be able to figure out exactly what is going on and what is being returned.

• Just as important, be consistent on your naming conventions for things. If you have a path / function where the input is environment and service, in another that has just environment, don't have the input be env and env_name. Keep them the same!

This is probably a very glossary approach, but this was the process i tried to follow as I created the one for my team.

This isn't exactly the post I planned on writing to break my writers block, but it's the one I needed.

I was listening to the podcast Linux Unplugged episode #395 and they had a little hacker challenge to see who could get root access with a non privileged user the quickest. The winning method was someone who realized the user had misconfigured access to run docker ps -a and mount / into a container to do gain the access to make the changes needed to complete the challenge.

Listening to this, I realized I'm just as guilty at configuring my docker access the same way. So I wanted to test the escalation method and play with the different methods to secure access to the docker socket (other than switching to Podman). The conventional recommendation to give access to docker is to give them access to the docker group that gets created with Docker's installation. This should really only be done with users who already have access to sudo and understand the risks that this entails. My team at work is guilty of giving access to the docker group on our sandbox server instead of using a proper method.

To prepare for this, lets set the scene:

There's a server inside a company that runs their company website using nginx and php in docker. There's a developer named Jeff who has an account on that server, but he has made the cardinal sin... His password is password. Jeff was granted access to the folder /data/website as well as the docker group to manage the website. His personal machine was compromised and a bad actor has used the bad password to gain access to the server.

The sysadmin for the server has also left a bunch of corporate passwords in a place they feel is safe, /root, but was smart enought to make sure the file was only available to the root user and nobody else.

Each of the examples will be done and shown on both debian and rhel based systems unless otherwise specified.

Lets go from easiest to less easy for things the bad actor can do with jeff's docker access. Lets steal the corporate passwords.

The bad actor got a tad lucky when mounting root and finding those passwords. They could have easily just mounted / and found that file sitting in /root.

This behavior is exactly what is expected when you create a container and mount a path. Docker has access to SUID, which grants any container with a mounted folder to read any file.

Gaining access to the system and mucking around isn't much more difficult, but depending on your server's base OS, the behavior is different.

By using chroot on the container you're able to get a root shell into the host system. Interestingly enough (was not aware of this prior to testing) the network stack isn't properly setup in the chroot shell so depending on things that are cached you may be able to install a package. Though That probably won't really stop someone trying to compromise your system.

But unfortunately that's not all someone can do with that chroot session. With it, they'd be able to create themselves new users, or even change the root password to give themselves easier access to the system.

See? jeff really didn't have permission.

Using the docker group, you are given an extremely wide berth to do nefarious things if you know what you are doing. But leveraging docker doesn't have to be all doom and gloom. There are ways to make modifications to your processs instead of just accepting this is the only way to do it.

For this next bit, we're only going to use our RHEL based system. We're going to remove jeff's access to the docker group and setup  ACL rules for him to interact with the docker socket. Unfortunately, this still has the privilege escalation problem and would allow you to much around with the system.

Depending on you're use case, which should always be taken into account when deciding access to tools, you may want to leave docker only available to root and then breaks the ability to user the docker group. This will require people to either be in the sudoers group (either passwordless or regular) to be able to interact with the docker socket. With this restriction, you can also then give specialized access with scripts to specific commands for people like jeff to be able to still work.

Docker now does have the option to run as rootless, but it's a bit more involved to get up and running. The next time you visit, I'll show how you can migrate your workflow from Docker to Podman, including using docker-compose.