Ran into some network issues and incredibly slow speeds. Here’s everything I tried in the most brute force manner available. It ended up being a setting in PFSense I needed to disable for Hardware Offloading.
Intro
It was raining outside and we had blown the fuse about two to three times. I really thought this was a router problem for the longest time. This was until I saw my roommantes PS5 was running close to 300MB/sec down. I thought to myself, why the F are the servers in the basement running so effing slow.
Every VM had incredibly slow speeds to the point of unusability. I was unable to ssh to any server leaving me to console in via the proxmox novnc interface. With most of these boxes being set up for serial connections, an hd view of what was going on was not an option, queue standard definition.
I was at first convinced someone in the house had rate-limited the connection to the mac of the switch but then realized no one was probably technical enough to do that, as well as that a quick check confirmed that wasn’t the case.
I was pretty stumped on the problem and didnt know where to start and what to look at so I started from the Bottom and worked my way up the stack.
1) Physical - Fine, cable looked alright, switch was having no problem 2) Ping - ICMP was about 5.00ms.. which is about normal for the house.. lead me to fw rules 3) Firewall Rules - Always check firewall Rules: Nothing changed, was paranoid for a few minutes I had been hacked and all the networks were slow for this reason - I thought ICMP was allowed through and ipv4 was.. 4) OS - I consoled into one of the VM’s and disabled IPV6 which I thought was an issue, it was not. 5) Applications were not working/they worked at a crawling speed.
Nothing I tried was working so I worked from outside the network in. 1) WAN - My laptop had wireless speeds that were fine. ~200mb/sec down 2) Router to Switch - This looked good - I didn’t really test the switch connection to the External WAN… I think a simple curl which came back in seconds confirmed I was fine 3) Switch to Physical Hardware - All the vlans were correct… No configuration change had happened and all the vlans were properly assigned. 4) Physical HW to Proxmox Interface - Again, nothing had really changed.. 5) Proxmox Interface to VM - Because PFSense is in a VM on the hardware I checked settings and speeds with IPERF3.. This is where I noticed the big discrepency..
So the PFSense boxes get their own WAN IP assignment.
This meant they can talk directly to the WAN. As an aside: because Proxmox was routed through the 10.0.0.0/24 network to the pfsense box to the WAN, I was even experiencing slow speeds from the physical servers too..
From the PFSENSE boxes though, they were seeing about 300mb/sec down..
This immedietly led me to a firewall configuration setting.
Hardware offloading
Hardware offloading as a quick definition is a way for functions to use dedicated hardware. In this case (the case of PFSense), this meant routing functions would have been sent to dedicated networking hardware. Becasue I don’t have dedicated networking hardware on the two servers this means the functions get wrapped a few times, go to the physical CPU, and then get executed. This would be as opposed to being called from the VM’s CPU.
From the pfsense virtual machine, there’s no sense in having offloading enabled. It slows down everything to a hault.
Easy enough fix:
System > Advanced > Networking > Hardware Checksum Offloading (check to disable)
System > Advanced > Networking > Hardware TCP Segmentation Offloading (check to disable)
System > Advanced > Networking > Hardware Large Receive Offloading (check to disable)