Stephen Newey

After a few months of getting comfortable with NixOS as my desktop operating system, I decided it was time to try it out for servers. But first I wanted to write about the setup I had before.

Disclaimer: This post is likely full of bad ideas – you probably shouldn't setup anything you care about like this. It is my opinion that my most valuable learning is when I'm learning what not to do, and I know there are some gems lurking in here.

How it started

At home, I was running a 8th-gen i5 NUC doing double duty as my home router and server, called homegw. It was running Ubuntu 22.04 with LXD. Router stuff happened on the base OS, with dnsmasq, AdGuard Home and a bunch of ufw rules. It also ran xl2tpd to bring up my Andrews and Arnold LT2P connection so I can have real IPv6 here, which my cable ISP doesn't provide.1

There was an LXD VM for Home Assistant, and LXD containers for a backup server (restic and Arq over SSH), for Plex, and one running various tools for... ahem downloading Linux ISOs.

There was a Wireguard tunnel from this machine out to Mullvad, and a static route with NAT to 10.64.0.1 on this link. Mullvad operates a SOCKS5 proxy on this address, so I configured all my Linux ISO downloading tools to use this proxy.

Another Wireguard tunnel connected to a dedicated server, hetty, running in Hetzner's Falkenstein data centre. It was nabbed from their server auction a little over a year ago. It was an 8-core 9th-gen i9, 128Gb RAM and 2x1Tb NVMe drives, for the bargain price of 54€ per month. In retrospect, way more computer than I needed, and more money than I should be spending.

This too ran Ubuntu 22.04, with LXD. In this case, a single LXD container called kubeservices running microk8s. I had Kubernetes setup with flux, a GitOps tool that keeps your cluster in sync with a bunch of YAML defined in Git. Hosted here were:

  • This blog (Writefreely with MySQL)
  • The CrunchyData Postgres Operator providing PostgreSQL instances for most of the other services mentioned here
  • My Mastodon instance, with Elasticsearch and Libretranslate
  • Vaultwarden, a full featured and self hosted Bitwarden server
  • Synapse, a Matrix chat server
  • Prometheus, for gather metrics on all the services, including from Home Assistant on the other end of the Wireguard tunnel, so I can get pretty graphs on my home electricity generation/usage, heating, etc.
  • Grafana for visualising those metrics
  • Keycloak for single sign-on to most of the above services

Kubernetes PersistentVolumeClaims provided all the stateful storage, using the microk8s host path provisioner, all ultimately ending up in a single directory on the host, an LXD custom volume with daily snapshots and backed up to OneDrive2 and to homegw with restic.

I'd chosen to use LVM on top of a LUKS device made from an mdadm initialised RAID1 mirror. ZFS was tempting, but whilst this was easy to setup with Ubuntu Desktop, it was less straightforward with Ubuntu Server. So I took the easy road. Or so I thought.

LXD creates a ThinPool for it's storage when using LVM, which eventually bit me when it's space reserved for metadata filled up. Despite having plenty of free data space, I couldn't figure out how to reallocate more space for metadata (I don't think you can). So I ended up ejecting the second SSD from the RAID1 mirror, and adding it as a new disk to the volume group in order to expand the size of the thin pool in order to recover.

That was my first big “this was a mistake” moment, letting LXD allocate all the remaining volume group space for a single Thin Pool was a bad idea. Ultimately I think LVM was not a good choice for this kind of setup, and I wouldn't make it again.

Oh, and I'd made the same choice of LVM with LXD on homegw. Multiple times I filled the disk and had a bit of a nightmare recovering from it. One does not simply.

Another “this was a mistake” moment was from filling OneDrive with restic backups. It turned out the Postgres Operator by default creates a single initial backup using pgrestbackup, and then captures the write-ahead log forever. Eventually my daily restic snapshots, even with regular pruning, filled the dedicated OneDrive account I'd setup for the job. At that point, restic could simply not recover. Any kind of pruning operation needed some amount of storage space in OneDrive that impossible to provide. You can pay for extra storage, but only on the primary Microsoft 365 account, so I couldn't buy my way out of it. In the end I trashed the entire restic repository and started again.

A simple change to the PostgresCluster YAML switched PGO to giving daily backups, storing up to 3 in the cluster. My disk usage went down to 600Gb to 150Gb after 3 days.

    pgbackrest:
      global:
        repo1-retention-full: "3"
        repo1-retention-full-type: count
      repos:
      - name: repo1
        schedules:
          full: "0 4 * * *"

Another consequence of filling the metadata for the thin pool mentioned earlier was the disk switched to read-only. Upon recovery I had some corrupted Postgres DBs and needed to restore from backup. The Postgres Operator makes that possible with by poking at different parts of the YAML. It's an incredible tool, but not knowing it inside out, I spent a lot of time feeling frustrated that I couldn't just jump on the server and fix things. Instead, everything is orchestrated, and it's feels a bit like operating a light switch with a broom stick.

For my use, I don't need anything like the capabilities PGO offers, and I should KISS.

The big takeaway

Understand your tools. Read the docs. Be curious about what can go wrong. Test those scenarios at your leisure, not in production.

Next time

All of that is gone. I'll be back to describe what replaced it.

1 Notably, I've found IPv4 performance is often better over this link too, with lower ping times to many sites with AA compared to Virgin Media, despite having to transit VM to get to AA first.

2 You can pick up Microsoft 365 Family for under 50GBP per year if you watch out for offers on the “gift card” version of it. This gives you 6x 1Tb OneDrive accounts, which is some of the cheapest cloud storage out there. Encrypting what you put there is a good idea, so tools like restic with rclone are your friends.

#hosting #kubernetes #ubuntu #linux

My home network is effectively in two segments, with the cable modem, router/server and access point wired together downstairs and another access point acting as a bridge, connected to a wired network in my office upstairs.

I'd been wanting to up my security game and split out dedicated networks (VLANs) and SSIDs for trusted devices, guests and untrusted IoT things for a while. One of the frustrating things about using Wi-Fi to bridge the two wired networks is that this typically precludes VLAN functionality.

I could just do things right and run a cable between the two wired islands, but my willingness to venture into technical escapades vastly exceeds my willingness to drill holes in walls and ceilings. And so, I spent a full day of my 2023 Christmas holiday messing around with OpenWrt.

All my routing is performed by my NUC server, so I'm using both OpenWrt devices purely as access points. Downstairs, I have a Ubiquiti Unifi 6 Long Range which was easy to reflash with vanilla OpenWrt by following the instructions on that link. Upstairs I have a Belkin RT3200, also flashed with vanilla OpenWrt. It turns out both devices are very similar, running the same SoC and Wi-Fi 6 chipsets. The Belkin unit has a 4 x 1GbE port switch + WAN port, where the Unifi has a single 2.5GbE uplink.

It is possible to pass VLANs over Wi-Fi by tunnelling your L2 network over IP, with the caveat of needing a higher than typically MTU to allow for the overhead. I created a mesh connection between the APs, bringing up network interfaces on either side with static IPs. Over that connection GRETAP devices on either end are able to pass Ethernet frames, and bridge to interfaces exposing endpoints for each VLAN.

I'm indebted to oofnikj for his post and more especially his config repo for getting me closer to a working configuration than the official docs.

If you need or want a similar setup, read on.

Tips

tcpdump was my friend on this adventure, revealing a storm of DHCP, ARP and multicast traffic at one point that was escaping the VLANs due to the switch in the Belkin unit effectively munging everything together and confusing the rest of the network until I got the configuration correct.

I started the configuration with both APs on the same wired network, next to each other. There came a point, just after bringing up the tunnel, where I had to make sure to disconnect one of them to avoid loopbacks. STP should take care of this, but didn't seem to for me, though that may have been down to the switch issue above.

Several times along the way I was saved by the dedicated tunnel network, where one AP was accessible, I could typically SSH from it using the internal IP assigned to the other to fix things.

My setup involves the trusted network behaving as the primary, untagged VLAN on each other physical Ethernet ports, with the IoT and Guest VLANs tagged.

Any feedback on what I could have done better is most welcome!

AP 1 Configuration (Unifi 6 LR, downstairs, attached to router)

# /etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd92:bb11:b3e9::/48'

# I'm not clear on why this is a device rather than an interface
# but it was the default config, and it works, so I don't feel like
# experimenting with it further now!
config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'
	# VLAN 1 on the interface connected to the GRE tunnel
	list ports '@trunk.1'
	option stp '1'
	option igmp_snooping '1'

# this interface holds the IP of the AP on the trusted network
config interface 'lan'
	option device 'br-lan'
	# I configure my IPs with fixed entries in DHCP
	option proto 'dhcp'
	# don't delegate IPv6, my router will take care of it
	option delegate '0'

# this interface is one end of the underlying IP network for the tunnel
# between the two APs
config interface 'wtun'
	option proto 'static'
	# any address outside our normal range is fine here
	option ipaddr '172.16.0.1'
	option netmask '255.255.255.0'
	option delegate '0'
	# it's here we need a larger-than-normal MTU to be able to pass
	# our 1500 MTU + overhead Ethernet frames
	option mtu '2048'

# the trunk bridge acts as the Ethernet endpoint of the GRE tunnel
# and we can create VLAN tagged versions of it to bridge to our
# wired and wireless interfaces
config interface 'trunk'
	option type 'bridge'
	option proto 'none'
	option bridge_empty '1'
	option delegate '0'
	option stp '1'
	option defaultroute '0'

# the GRE tunnel itself
config interface 'gre'
	option proto 'gretap'
	option ipaddr '172.16.0.1'
	option peeraddr '172.16.0.2'
	# transit over the dedicated network we created
	option tunlink 'wtun'
	# attach to our trunk bridge
	option network 'trunk'
	option df '0'
	option mtu '1500'
	option delegate '0'

# a bridge connecting the IoT (101) VLAN between the GRE
# tunnel and whatever we want to attach to it
config device
	option name 'br-iot'
	option type 'bridge'
	list ports 'br-lan.101'
	list ports '@trunk.101'
	option stp '1'
	option igmp_snooping '1'

# bring up an interface on the IoT VLAN for this AP
config interface 'iot'
	option proto 'dhcp'
	option device 'br-iot'
	option defaultroute '0'
	option peerdns '0'
	option delegate '0'

# a bridge connecting the Guest (102) VLAN between the
# GRE tunnel and whatever we want to attach to it
config device
	option name 'br-guest'
	option type 'bridge'
	list ports 'br-lan.102'
	list ports '@trunk.102'
	option stp '1'
	option igmp_snooping '1'

# bring up an interface on the Guest VLAN for this AP
config interface 'guest'
	option proto 'dhcp'
	option device 'br-guest'
	option defaultroute '0'
	option peerdns '0'
	option delegate '0'

AP2 Configuration (RT3200, upstairs)

# /etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fddd:12af:6646::/48'

config interface 'wtun'
	option proto 'static'
	option ipaddr '172.16.0.2'
	option netmask '255.255.255.0'
	option delegate '0'
	option mtu '2048'

config interface 'trunk'
	option type 'bridge'
	option proto 'none'
	option bridge_empty '1'
	option delegate '0'
	option stp '1'
	option defaultroute '0'

config interface 'gre'
	option proto 'gretap'
	option ipaddr '172.16.0.2'
	option peeraddr '172.16.0.1'
	option tunlink 'wtun'
	option network 'trunk'
	option df '0'
	option mtu '1500'

# my lan bridge is a little different on this AP due to the switch
# ports available on this device, which allow me to dedicate specific
# ports to VLANs as needed. Here every port is configured with the
# trusted network (VLAN 1) as it's untagged, primary VLAN and the
# other networks are available with tags on each port.
config device
	option name 'br-lan'
	option type 'bridge'
	option stp '1'
	option igmp_snooping '1'
	option bridge_empty '1'
	# attach VLAN 1 of the tunnelled Ethernet to this bridge, doesn't
	# seem quite right, but breaks without it
	list ports 'br-trunk.1'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'
	list ports 'wan'

# the untagged primary trusted VLAN on all ports
config bridge-vlan
	option device 'br-lan'
	option vlan '1'
	list ports 'br-trunk.1:u*'
	list ports 'lan1:u*'
	list ports 'lan2:u*'
	list ports 'lan3:u*'
	list ports 'lan4:u*'
	list ports 'wan:u*'

# tag the IoT VLAN on all ports
config bridge-vlan
	option device 'br-lan'
	option vlan '101'
	list ports 'lan1:t'
	list ports 'lan2:t'
	list ports 'lan3:t'
	list ports 'lan4:t'
	list ports 'wan:t'
	option local '0'

# tag the Guest VLAN on all ports
config bridge-vlan
	option device 'br-lan'
	option vlan '102'
	list ports 'lan1:t'
	list ports 'lan2:t'
	list ports 'lan3:t'
	list ports 'lan4:t'
	list ports 'wan:t'

# bring up an interface for the AP on the trusted VLAN
config interface 'lan'
	option device 'br-lan.1'
	option proto 'dhcp'
	option delegate '0'

config device
	option name 'br-iot'
	option type 'bridge'
	option stp '1'
	option igmp_snooping '1'
	list ports 'br-lan.101'
	list ports 'br-trunk.101'

config interface 'iot'
	option proto 'dhcp'
	option ipv6 '0'
	option device 'br-iot'
	option defaultroute '0'
	option peerdns '0'
	option delegate '0'

config device
	option name 'br-guest'
	option type 'bridge'
	option stp '1'
	option igmp_snooping '1'
	list ports 'br-lan.102'
	list ports 'br-trunk.102'

config interface 'guest'
	option proto 'dhcp'
	option ipv6 '0'
	option device 'br-guest'
	option defaultroute '0'
	option peerdns '0'
	option delegate '0'

Common configuration

The wireless and firewall configurations are identical on both APs. If your APs have different hardware, you'll likely need to adjust the radio sections for each.

# /etc/config/wireless
config wifi-device 'radio0'
	option type 'mac80211'
	option path 'platform/18000000.wmac'
	option channel '6'
	option band '2g'
	option htmode 'HT20'
	option cell_density '0'

config wifi-device 'radio1'
	option type 'mac80211'
	option phy 'wl1'
	option country 'US'
	option cell_density '0'
	option htmode 'HE80'
	option band '5g'
	option channel '149'

# the mesh network underlying our GRE tunnel
# I'm only using the 5GHz radio for this, because the signal is strong
# and the access points are static
config wifi-iface 'wifinet7'
	option device 'radio1'
	option mode 'mesh'
	option encryption 'sae'
	option mesh_id 'upstream'
	option mesh_fwding '1'
	option mesh_rssi_threshold '0'
	option key 'SomeRandomString'
	option network 'wtun'
	option ifname 'wtun'

# the 2GHz version of my trusted network SSID
config wifi-iface 'wifinet2'
	option device 'radio0'
	option mode 'ap'
	option ssid 'Trusted'
	option encryption 'sae'
	option key 'SuperStrongPassphrase'
	option network 'lan'

# the 5GHz version of my trusted network SSID
config wifi-iface 'wifinet5'
	option device 'radio1'
	option mode 'ap'
	option ssid 'Trusted'
	option encryption 'sae-mixed'
	option key 'SuperStrongPassphrase'
	option network 'lan'

# the 2GHz version of my Guest network SSID
config wifi-iface 'wifinet9'
	option device 'radio0'
	option mode 'ap'
	option ssid 'Guest'
	option encryption 'sae-mixed'
	# setting isolate to prevent wireless devices on this
	# SSID from talking to each other (same on IoT)
	option isolate '1'
	option key 'welcomeguest'
	option network 'guest'

# the 5GHz version of my Guest network SSID
config wifi-iface 'wifinet8'
	option device 'radio1'
	option mode 'ap'
	option ssid 'Guest'
	option encryption 'sae-mixed'
	option key 'letmeinplease'
	option network 'welcomeguest'
	option isolate '1'

# the 2GHz version of my IoT network SSID
config wifi-iface 'wifinet10'
	option device 'radio0'
	option mode 'ap'
	option ssid 'IoT'
	option encryption 'psk2'
	option isolate '1'
	option key 'untrusted'
	option network 'iot'

# the 5GHz version of our my network SSID
config wifi-iface 'wifinet11'
	option device 'radio1'
	option mode 'ap'
	option ssid 'IoT'
	option encryption 'psk2'
	option isolate '1'
	option key 'untrusted'
	option network 'iot'
# /etc/config/firewall
config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

# I have a firewall zone for each network
config zone
	option name 'lan'
	# only the trusted network and underlying tunnel network
	# have "input 'ACCEPT'" which will allow access to the
	# AP admin interfaces
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config zone
	option name 'guest'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'guest'

config zone
	option name 'iot'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'iot'

config zone
	option name 'tunnel'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'wtun'

Updated 2023-11-13 with Preview.app features, prompted by Neil's helpful response Desktop Linux: the software I'm currently using

With my new laptop coming in a few days, I'm finally thinking through the implications of moving away from macOS whilst still having an iPhone. These are some questions I need to answer:

  • How will I listen to music?
    • Currently streaming with Apple Music, blended by the Music app with my local music collection
  • How will I manage photos?
    • Currently in Photos, synced with iCloud, mostly originating from my phone, with a collection of old stuff originally synced from the Mac
  • How will I do quick annotations on screenshots, markup PDFs, resize/crop/export images without Preview.app?
  • What Passkeys do I need to migrate out of iCloud (or the Mac specifically)?
  • What other credentials are in my Mac/iCloud keychain that aren't in 1Password for some reason?
  • Safari has been my primary browser and macOS, and will likely continue to be on iOS
    • I tend to throw stuff at Reading List to pick up later, which I won't have access to on my new computer, so what to do instead?
  • Should I keep an old Mac around, running?
    • Logged into iCloud, it could provide another authentication factor for iDevices, which assume you've got some other Apple device nearby to do authentication
    • There are bridges for iMessage that could let me continue to read/reply to iMessage and SMS from my phone
    • I have an old MacBook Pro with a non-functioning screen that's too expensive to fix that could do the job
  • I subscribe to Microsoft 365, mainly for cheap cloud storage (effectively 6Tb for around ~£45 per year when bought on semi-frequent offer), but do use Excel for a few tasks. Should I...
    • Migrate away to something else?
    • Use the web version?
    • Try Wine/Crossover/Windows VM?
  • I almost entirely interact with Mastodon through Ivory, on my phone and with the macOS app. I really like that it stays in sync across both with my read position. Am I just going to use the iOS app now, or is there some other solution?

I guess I'll find my answers in the coming weeks.

#macos #linux

That whole writing more thing went well, didn't it? ?

Let's try a little brain dump of the nerdy things I've assigned myself to do:

  • Replace macOS with NixOS as my primary computing experience.
    • Being a cloud infrastructure engineer, I love me some declarative configuration.
    • Apple annoyed me with the offer of a £700 fix for the most expensive computer I ever purchased that was just out of warranty and had display issues.
    • The £700 was to replace a display that's screwed because the connecting cable has been damaged by their hinge design. There is a company in London that offer a £300 repair. Still quite ouchy.
    • It's an Intel Mac, how long are they even likely to support it anyway?
    • It was already passed on to Sean, replaced for me by a 14” M1 Pro.
    • Framework seems kinda great, so I'm eagerly awaiting my Ryzen-based Framework 13. At which point Sean can have the 14” and become untethered from a desk again.
    • I'll miss the screen of the MacBook, but I think even more I'll miss the speakers.
  • Capture my (non-phone) computing world into a Git repo.
    • Nix allows me to describe my systems and my user environments in code!
    • On that whole phone thing, wouldn't it be nice if there were some genuinely open-source phone ecosystem without Google that could actually run the apps I've come to depend on (banking, etc)?
    • I've sadly accepted my iPhone 12 mini will eventually be replaced with another iPhone.
  • Adopt a more keyboard-centric computing life, with a tiling window manager.
    • Endless configuration tweaking awaits.
  • Have a go at (neo)vim being my primary editor again, or otherwise invest properly in VSCod(e|ium).
    • I might as well go all in, eh?
    • Failing at this and continuing to use GoLand and PyCharm wouldn't be terrible.
  • Migrate my Hetzner dedicated server running this site, and my Masto instance to something else (self-host, Hetzner cloud).
    • It's way over-specced for my current use (but 50€ for 128GB RAM, 2x1Tb SSD and 8-core i9-9900K is amazing value).
    • I use microk8s on it as a single node instance, which is pretty inflexible, storage being a large part of sense of unease with that.
    • I made the mistake of choosing LVM and let LXD create a thin-pool consuming 100% of the remaining storage.
    • It ran out of metadata space, and you can't grow it into the unused data space, so I had to de-RAID1 the SSDs to make things work so that's a bit of a mess now.
    • Said mess returns errors when trying to do operations like take snapshots for backups, or even delete old snapshots, a reboot might fix it, or it might leave me with a complex failed boot to resolve.
    • Hetzner Cloud seems to be pretty darn cheap, and I can create a proper Kubernetes environment with separate control plane and real storage volumes and come in at similar or lower price than the dedicated server, albeit with less overall resources and with non-dedicated CPU.
    • I experimented briefly with Talos Linux on it yesterday, and it went pretty smoothly.
    • I can Terraform it (or OpenTF eventually, right?)
    • Or maybe I can just host it all at home on a single box?
  • Migrate my home server from Ubuntu with LXD to NixOS with something.
    • It's an 8th gen NUC in a fanless Asaka case.
    • It's my router and adds IPv6 to my home Internet with AAISP's L2TP service, because Virgin Media can't even, but I'm definitely taking their 1Gb/100Mb service over my next best option of 40Mb/10Mb DSL.
    • It's running an LXD-launched VM for Home Assistant, and LXD containers for Plex, some *aars, etc.
    • It's also built on the same LVM thin pool scheme as the Hetzner box, so is ultimately doomed.
    • I'm clearly going to NixOS it, and microvms looks super interesting.

I can't help think of Amelie's father and his toolbox.

#nixos #linux #cloud

I want to improve my writing, so I need to do more of it.

Improving means that I can communicate effectively, efficiently and engagingly. It means you can understand me, that I'm not wasting your time, and that I can hold your attention long enough to impart what I wanted you to know.

Part of my job as a software engineer is to express ideas in a way that is understandable by machines. More importantly, those ideas can be understood by other engineers who work with me now and in the future, whether I'm present or not. That is what good code means to me.

When I write for a machine I'm able to get feedback near instantly. Frequently that feedback is unambiguous. I was either successful in communicating my intent, or I was not. Code reviews help ensure I'm successfully communicating with other engineers too.

Much of my work is useful to people who shouldn't need to understand the details of it’s construction to find it helpful. I should be able to describe it in a way that respects their time and attention. Documentation matters.

Career progression requires I am able to communicate ideas in a way that places them in a wider context to demonstrate their value. I distill the ideas of multiple people and in doing so am representing them as well as myself.

I need to write to justify advancements, to mediate conflict, to convince others to invest their resources, to give candid feedback kindly, to explain failure, and to celebrate success.

I need to be able to do all of that in a timely manner that leaves room for all the other things to be done. Improving also means writing more quickly when that's necessary.

Success will be hard to measure here. Improvement needs feedback as well as practice. That's why I'm writing this publicly.

The most uncomfortable part is convincing myself that this is good enough, when I know that it could be better. If the only outcome is that getting easier, then this will be worthwhile.

#writemore