As I was reviewing SSH logs in Elasticsearch for my Gitea server I noticed quite a few failures all coming from a local address (192.168.1.0/24). This immedietly threw a red flag for me as I realized HAPROXY wasn’t forwarding the real IP to the SSH server within the Gitea Container. I came to find out SSH doesn’t support proxying and tcp doesn’t really support the proxy-protocol, but there is a tool called mmproxy that can “translate” the proxy-protocol. This blog post covers the configuration changes I made to HAPROXY, my Container running the SSH server, and the go-mmproxy configurations I made to support the proxy-protocol.
Move SSH to Port 10022 on Cloud Server
I moved the SSH port on the public reverse proxy to 10022 to support HAPROXY to use port 22. I may disable remote SSH login all together for security, but right now I plan to leave it.
This was done with the following
Include /etc/ssh/sshd_config.d/*.conf Port 10022 # <-- This line was uncommented and the port was changed #AddressFamily any #ListenAddress 0.0.0.0 #ListenAddress ::
After this change was made I restarted ssh, opened up the firewall for port 10022 and was able to verify logging in.
Of Note I also checked to confirm nothing was using port 22 with
Configuring HAPROXY to use port 22 and setting up the reverse proxy for TCP Connections on Port 22
My HAPROXY is configured like the following (note this is not the full configuration and I also use nginx for web traffic proxying)
defaults log global mode tcp # The rest excluded for brevity... resolvers resolver1 # nameserver internal.ip.dns.ipaddress:53 nameserver 22.214.171.124:53 126.96.36.199:53 frontend TCPFE bind :22 # <- Note the bind is on port 22 mode tcp # <- Note the mode is TCP timeout client 30s acl acl_gitea_port22 dst_port eq 22 use_backend gitea-ssh if acl_gitea_port22 backend gitea-ssh mode tcp balance source stick-table type ip size 50k expire 30m stick on src timeout connect 30s timeout server 30s server gitea-22 hrvgitp1.cell.jmoore53.com:2223 send-proxy # <- Note for this the port is 2223 and the send-proxy is also included
Port 2223 is the port where mmproxy will be running.
For HAPROXY to run on port 22 I had to modify
/etc/sysctl.conf configuration to include the following:
Then I ran:
sysctl -p and
systemctl restart haproxy which allowed HAPROXY to bind to port 22
Configuring Gitea to use host ports 2222 and 3000
I ran into an odd issue when configuring mmproxy and docker where the proxy-protocol would be stripped at the virtual machine level, but since there was one more level of NAT from the virtual machine to the docker container the logs in Gitea would show the docker gateway IP as the source. The fix was for me to use host networking for the docker container ports. I could have decided to modify the dockerfile entrypoint.sh to run yet another service within the container, but decided just to exposet the ports on the virtual machine level.
Within the Container
app.ini I configured the container to use port 2222 to not overlap the openssh port for the gitea virtual machine:
[server] APP_DATA_PATH = /data/gitea DOMAIN = gitea.jmoore53.com SSH_DOMAIN = gitea.jmoore53.com HTTP_PORT = 3000 ROOT_URL = https://gitea.jmoore53.com/ DISABLE_SSH = false SSH_PORT = 22 # <- This is the port that shows on the website SSH_LISTEN_PORT = 2222 <- This is the actual port ssh is using when it runs on the container OFFLINE_MODE = false
I updated the docker-compose.yml file to look like the following:
version: "3" networks: host: external: true services: server: image: gitea/gitea:1.20.5 container_name: gitea environment: - USER_UID=1000 - USER_GID=1000 - SSH_LISTEN_PORT=2222 # <- Also passed this into the container as an environment variable restart: always network_mode: host <- This is the needed line volumes: - ./gitea-data:/data - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro ports: - "3000:3000" # <- These aren't used because the container's networking is in host mode - "2222:2222" # <- These aren't used because the container's networking is in host mode
Setting up go-mmproxy
Getting go-mmproxy setup was very easy considering all my traffic is within one virutal machine. Routing can be made more complex using different virtual machines, but all the traffic was over the
lo device. The routing changes required were the following:
ip rule add from 127.0.0.1/8 iif lo table 123 ip route add local 0.0.0.0/0 dev lo table 123 ip -6 rule add from ::1/128 iif lo table 123 # <- these are to support ipv6 (currently not using) ip -6 route add local ::/0 dev lo table 123 # <- these are to support ipv6 (currently not using)
After the routing rules were set, I ran
go-mmproxy with the following, note there’s no IPV6:
sudo go-mmproxy -l 0.0.0.0:2223 -4 127.0.0.1:2222
With this command go-mmproxy binds on port 0.0.0.0:2223 (note this is the port HAPROXY is sending tcp traffic to) and then forwarding traffic to 127.0.0.1:2222 where gitea is running. Go-proxy removes the proxy-protocol and we are then able to see the real ip address within the gitea logs.
Creating a Service for go-mmproxy on the Gitea service
Modify the following file
#!/bin/bash ip rule add from 127.0.0.1/8 iif lo table 123 ip route add local 0.0.0.0/0 dev lo table 123 ip -6 rule add from ::1/128 iif lo table 123 # <- these are to support ipv6 (currently not using) ip -6 route add local ::/0 dev lo table 123 # <- these are to support ipv6 (currently not using) go-mmproxy -l 0.0.0.0:2223 -4 127.0.0.1:2222
Then modify the following service file
[Unit] Description=Gitea go-mmproxy After=network.target StartLimitIntervalSec=0 [Service] Type=simple Restart=always RestartSec=1 User=root ExecStart=/usr/local/sbin/goproxy.sh [Install] WantedBy=multi-user.target
After the service is created, reload systemctl-daemons and enable and start the service:
sudo systemctl daemon-reload sudo systemctl enable go-mmproxy.service sudo systemctl start go-mmproxy.service