Finding unity in OpenVPN and macvlan

Brief intro on my setup.

I have a small server that I run pretty much everything out of. It's been a personal playground for about 4-5 years now. To try and keep the base system clear from mistakes and stupidity, I run things in lxc containers. I started with them back when docker was still new and it didn't make much sense past normal development work. That's changed recently and my way of interfacing with my server has changed because of it.

I originally had OpenVPN running in a container, but a kernel upgrade broke that and instead of trying to figure out what was going on, or which Kernel was still stable, I moved the VPN from a container to the host system. Problem was, I have everything in the lxc containers and I have them getting IPs on my network to make dealing with them easier. To do this I work with the macvlan config and the limitation is that the host cannot talk to the guests directly, and vise versa.

I've been living with this issue for a while, and with me moving to playing with docker and eventually attempting kubernetes in same server, I need to fix that to make the applications available while on the vpn.

MACVLAN AND LXC ARE WONDERFUL

Honestly, the 2 work seamlessly together, but the host<->guest limitation is annoying.

There's lots of things out there that say "just do this and it works". That's great and all, but I want to understand what's going on with the commands. Unfortunately good blog posts about it either don't exist anymore or never got shared past the commands to get it working.

For this to work, you need to be on an OS that uses ifupdown, or switch to it away from Netplan.

Part 1: Edit your interfaces

vi /etc/network/interfaces

auto macvlan0
iface macvlan0 inet static
address 192.168.2.6
netmask 255.255.255.0
pre-up route del default
pre-up route del -net 192.168.2.0 netmask 255.255.255.0
pre-up ip link add link bond0 name macvlan0 type macvlan mode bridge
gateway 192.168.2.1

So what's going on here is that we need to add a bridged interface at the host level for our lxc network config to bond to (not bond0, that's my main interface on this server). Basically what's happening is that the macvlan that you configure in the profile is running at the application level and forwarding the requests to the host. We need a place for the macvlan to go. If we create a virutal interface for LXD to bond to, we'll be able to route the traffic properly.

Biggest difference with the above compared to what I found in archive.org was defining the address and netmask. For some reason the system would not take the interface without them defined. I lose an IP, but it's worth the cost.

restart networking

Part 2: Edit the profile for the containers you want to ssh into.

lxc stop $container_names_here

lxc profile edit default

description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: macvlan0
    type: nic
name: default

Here is how we have the profile configured. Notice the line parent. I changed mine from bond0 to macvlan0. that changes where and how it bonds to the nic config on the host.

Once saved, start the containers and test.

[nagel@gizmo lxd]$ lxc list  | grep docker -A 1 | grep eth0
|                  |         | 10.1.34.81 (eth0)              |      |            |           |
[nagel@gizmo lxd]$ ssh 1923.168.2.81
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.4.0-170-lowlatency x86_64)

 ...

Last login: Mon Feb 24 23:54:45 2020 from 10.1.34.6
nagel@docker:~$

Hooray!

OpenVPN routing woes

It wasn't until a little bit later that I realized that I made a mistake without realizing it. I bork my OpenVPN config. I could make the initial connection, but the TLS handshake would fail.

It took a little sleuthing, but I realized it was because of  server.conf setting and the macvlan. Traffic was making it into bond0 just fine, but wasn't leaving out of a port that had external access. My server.conf file had this line at the very top ;local a.b.c.d. Basically, listen wherever there's an IP. I had to change that line to read local 192.168.2.5 instead. Forcing it to read only from the IP that is configured to leave my router.