As I was reviewing SSH logs in Elasticsearch for my Gitea server I noticed quite a few failures all coming from a local address (192.168.1.0/24). This immedietly threw a red flag for me as I realized HAPROXY wasn’t forwarding the real IP to the SSH server within the Gitea Container. I came to find out SSH doesn’t support proxying and tcp doesn’t really support the proxy-protocol, but there is a tool called mmproxy that can “translate” the proxy-protocol. This blog post covers the configuration changes I made to HAPROXY, my Container running the SSH server, and the go-mmproxy configurations I made to support the proxy-protocol.
Move SSH to Port 10022 on Cloud Server
I moved the SSH port on the public reverse proxy to 10022 to support HAPROXY to use port 22. I may disable remote SSH login all together for security, but right now I plan to leave it.
This was done with the following /etc/ssh/sshd_config:
Include /etc/ssh/sshd_config.d/*.conf
Port 10022 # <-- This line was uncommented and the port was changed
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
After this change was made I restarted ssh, opened up the firewall for port 10022 and was able to verify logging in.
Of Note I also checked to confirm nothing was using port 22 with netstat:
netstat -tunlp
Configuring HAPROXY to use port 22 and setting up the reverse proxy for TCP Connections on Port 22
My HAPROXY is configured like the following (note this is not the full configuration and I also use nginx for web traffic proxying) /etc/haproxy/haproxy.cfg:
defaults
log global
mode tcp
# The rest excluded for brevity...
resolvers resolver1
# nameserver internal.ip.dns.ipaddress:53
nameserver 8.8.8.8:53 8.8.8.8:53
frontend TCPFE
bind :22 # <- Note the bind is on port 22
mode tcp # <- Note the mode is TCP
timeout client 30s
acl acl_gitea_port22 dst_port eq 22
use_backend gitea-ssh if acl_gitea_port22
backend gitea-ssh
mode tcp
balance source
stick-table type ip size 50k expire 30m
stick on src
timeout connect 30s
timeout server 30s
server gitea-22 hrvgitp1.cell.jmoore53.com:2223 send-proxy # <- Note for this the port is 2223 and the send-proxy is also included
Port 2223 is the port where mmproxy will be running.
For HAPROXY to run on port 22 I had to modify /etc/sysctl.conf configuration to include the following:
net.ipv4.ip_nonlocal_bind=1
Then I ran: sysctl -p and systemctl restart haproxy which allowed HAPROXY to bind to port 22
Configuring Gitea to use host ports 2222 and 3000
I ran into an odd issue when configuring mmproxy and docker where the proxy-protocol would be stripped at the virtual machine level, but since there was one more level of NAT from the virtual machine to the docker container the logs in Gitea would show the docker gateway IP as the source. The fix was for me to use host networking for the docker container ports. I could have decided to modify the dockerfile entrypoint.sh to run yet another service within the container, but decided just to exposet the ports on the virtual machine level.
Within the Container app.ini I configured the container to use port 2222 to not overlap the openssh port for the gitea virtual machine:
[server]
APP_DATA_PATH = /data/gitea
DOMAIN = gitea.jmoore53.com
SSH_DOMAIN = gitea.jmoore53.com
HTTP_PORT = 3000
ROOT_URL = https://gitea.jmoore53.com/
DISABLE_SSH = false
SSH_PORT = 22 # <- This is the port that shows on the website
SSH_LISTEN_PORT = 2222 <- This is the actual port ssh is using when it runs on the container
OFFLINE_MODE = false
I updated the docker-compose.yml file to look like the following:
version: "3"
networks:
host:
external: true
services:
server:
image: gitea/gitea:1.20.5
container_name: gitea
environment:
- USER_UID=1000
- USER_GID=1000
- SSH_LISTEN_PORT=2222 # <- Also passed this into the container as an environment variable
restart: always
network_mode: host <- This is the needed line
volumes:
- ./gitea-data:/data
- /etc/timezone:/etc/timezone:ro
- /etc/localtime:/etc/localtime:ro
ports:
- "3000:3000" # <- These aren't used because the container's networking is in host mode
- "2222:2222" # <- These aren't used because the container's networking is in host mode
Setting up go-mmproxy
I made an attempt at running the regular mmproxy proxy, however ran into some issues building it with libtools. I found the go-mmproxy and an apt package for go-mmproxy which I decided to use.
Getting go-mmproxy setup was very easy considering all my traffic is within one virutal machine. Routing can be made more complex using different virtual machines, but all the traffic was over the lo device. The routing changes required were the following:
ip rule add from 127.0.0.1/8 iif lo table 123
ip route add local 0.0.0.0/0 dev lo table 123
ip -6 rule add from ::1/128 iif lo table 123 # <- these are to support ipv6 (currently not using)
ip -6 route add local ::/0 dev lo table 123 # <- these are to support ipv6 (currently not using)
After the routing rules were set, I ran go-mmproxy with the following, note there’s no IPV6:
sudo go-mmproxy -l 0.0.0.0:2223 -4 127.0.0.1:2222
With this command go-mmproxy binds on port 0.0.0.0:2223 (note this is the port HAPROXY is sending tcp traffic to) and then forwarding traffic to 127.0.0.1:2222 where gitea is running. Go-proxy removes the proxy-protocol and we are then able to see the real ip address within the gitea logs.
Creating a Service for go-mmproxy on the Gitea service
Modify the following file /usr/local/sbin/goproxy.sh:
#!/bin/bash
ip rule add from 127.0.0.1/8 iif lo table 123
ip route add local 0.0.0.0/0 dev lo table 123
ip -6 rule add from ::1/128 iif lo table 123 # <- these are to support ipv6 (currently not using)
ip -6 route add local ::/0 dev lo table 123 # <- these are to support ipv6 (currently not using)
go-mmproxy -l 0.0.0.0:2223 -4 127.0.0.1:2222
Then modify the following service file /etc/systemd/system/go-mmproxy.service:
[Unit]
Description=Gitea go-mmproxy
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=root
ExecStart=/usr/local/sbin/goproxy.sh
[Install]
WantedBy=multi-user.target
After the service is created, reload systemctl-daemons and enable and start the service:
sudo systemctl daemon-reload
sudo systemctl enable go-mmproxy.service
sudo systemctl start go-mmproxy.service