Not really what I wanted, but I got there eventually.
So the virtual firewall is a great idea until the host goes down and everything routes through the firewall.
Unfortunately - Proxmox’s HA is “Wait until the host is back online and power the VM back on” as compared to a true HA where the VM is on a hot standby on the other physical node. This was a little disappointing to figure out, hence the need for redundant firewalls that each exist on two hosts.
For a little context, everything in my homelab runs through the firewall. VLANs are configured so everything is within a 10.0.0.0/8 network, this includes hosts and ilo’s. The only way to access these devices is through physical connection or the firewall.
Mistake 1, Routing Broken!
Yep - So I didn’t realize that the Proxmox hosts needed an IP on the home LAN (192.168.200.0/24 network) for the bridge in Proxmox to work. So I moved the IP of the Proxmox host and everything broke, Fun!
This meant I couldn’t access anything. Because the host didn’t have an IP anymore, that meant the firewall didn’t get one through the bridge. Everything was offline, enter me taking a workstation to the lab, assigning a vlan tag and going after it.
Manjaro! XORG Broken! Fun
Yeah - so I had a crazy uptime going on 90ish days pulling rolling updates for Manjaro. Who would have ever guessed xorg broke when I finally rebooted.
Ended up being a really dumb vnc config file for Xorg for remote vnc sessions. Nonetheless, spent a good hour trying to figure this one out digging through journal logs and system logs.
Anyway, this manjaro workstation has two nic’s, a wireless, and a wired. I connected to wireless from my laptop via ssh, then enabled the vlan10 (10.0.0.0/24) for the wired connection into the switch where everything is.
Because the default gateways were broken I only had access to the local subnet, which ended up being fine because I was able to connect to everything I needed.
Opened up a proxy for my laptop and viola I was able to connect to all the management devices again.
Whatever, I fix the proxmox issues and I’m back where I wanted to be.
Double Trouble
Already had one FW spun up, that’s what I was using for my “current” configuration. This time All I had to do was spin up another with basically the same config.
Easy enough.
Both were online.. well.. Ok so on my home lan, 192.168.200.1/24 is the router, fine, .2 is the switch to my homelab, good, .3 is hlpmx1 (physical proxmox server), .4 is hlpmx2 (physical proxmox server), .37 was assigned via dhcp to the hlvfw1 (virtual firewall).
Well… Smart me decided to move the first firewall to .3 and build the second firewall to .4.. Static addressing and conflicts! Fun! More headaches!
Ended up moving the firewalls to .6 and .7 on the network. .5 is the virtual IP.
After this mess, both firewalls were up and everything was looking good.
Sync VLAN
Created a VLAN for just syncing. It worked as expected. It’s a vlan. nothing really crazy going on.
Syncing, but to the wrong node
Ended up being easy, both Pfsense boxes were on the same version.
I created a Sync user with HA abilities and a complex password.
Get this. I synced both firewalls to the wrong node. Instead of syncing P2’s config to P1, I synced P1’s configuration to P2.
This meant basically all rules were wiped. Should have made a backup, but next time I’ll know. (There weren’t that many rules anyway so it didn’t really matter)
CARP Address
Setting up CARP was a snap… I set up a CARP Address for each interface, ended up picking the top of the range (10.0.0.254 as an example).
OpenVPN Configuration with CARP IP
After setting the CARP IP make sure to update any WAN settings where the firewall interface may be lingering. Walking through the OpenVPN wizard you will need to configure the VPN to run on the CARP IP instead of the WAN Interface. This got me..
I was running the OpenVPN server on the firewall2 WAN interface causing a heck of a lot of problems, but packet filter, and logs led me to where I needed to fix.
Just needed to update the openvpn server config to use the CARP IP.
Lessons learned
- Backup before really breaking things
- Make sure you have a confirmed connection before introducing breaking changes
- Interfaces in OpenVPN