Setting up a custom indexer using a Distributed Hash Table (DHT) requires careful configuration and deployment. This post outlines the steps taken to install bitmagnet on an Alpine Linux virtual machine (VM), configure OpenVPN for secure tunneling, and use NFS for storage instead of MinIO due to compatibility issues.
Installing DHT on Alpine VM
To begin, a new Alpine VM was set up to run the DHT. Transmission and OpenVPN were also installed on this server. Here is the output of the ip route command after configuring the network:
alpineylinux:~# ip route
0.0.0.0/1 via 10.35.112.1 dev tun0
default via 192.168.122.1 dev eth0 metric 202
10.35.112.0/24 dev tun0 proto kernel scope link src 10.35.112.226
10.100.0.0/24 via 192.168.122.1 dev eth0
128.0.0.0/1 via 10.35.112.1 dev tun0
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
172.20.0.0/16 dev br-94bb901b6c16 proto kernel scope link src 172.20.0.1
172.23.0.0/16 dev br-716551f713d9 proto kernel scope link src 172.23.0.1 linkdown
191.96.150.41 via 192.168.122.1 dev eth0
192.168.122.0/24 dev eth0 proto kernel scope link src 192.168.122.115
Configuring OpenVPN
OpenVPN was configured to provide a secure tunnel interface on the VM. The key configuration file, openvpn.cnf, is shown below:
client
dev tun0
proto udp
remote us-newyorkcity.privacy.network 1198
resolv-retry infinite
nobind
persist-key
persist-tun
cipher aes-128-cbc
auth sha1
tls-client
remote-cert-tls server
auth-user-pass /etc/openvpn/creds.conf
compress
verb 1
reneg-sec 0
redirect-gateway def1
route 10.100.0.0 255.255.255.0 net_gateway
<crl-verify>
-----BEGIN X509 CRL-----
-----END X509 CRL-----
</crl-verify>
<ca>
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
</ca>
disable-occ
The most important line in this configuration is route 10.100.0.0 255.255.255.0 net_gateway, which instructs the VM to route all traffic to the 10.100.0.0 network using the local network gateway.
Docker Configuration for PostgreSQL
To manage data, a Docker container running PostgreSQL was set up with the following docker-compose.yml configuration:
postgres:
image: postgres:16-alpine
container_name: bitmagnet-postgres
volumes:
- ./data/postgres:/var/lib/postgresql/data
# ports:
# - "5432:5432" Expose this port if you'd like to dig around in the database
restart: unless-stopped
environment:
- POSTGRES_PASSWORD=postgres
- POSTGRES_DB=bitmagnet
- PGUSER=postgres
shm_size: 1g
healthcheck:
test:
- CMD-SHELL
- pg_isready
start_period: 20s
interval: 10s
NFS was used for storing the data due to issues with using s3fuse directories with Transmission. This setup runs on a VM with 8 GB of storage, 1 GB of RAM, and 1 vCPU. Although using an Alpine container could further reduce resource usage, the VM remains a more straightforward solution due to concerns about routing and VPN connections within Docker containers.
Links
- bitmagnet setup and installation
- bitmagnet endpoints
- bitmagnet internals and development
- bitmagnet GitHub repository
- bitmagnet DHT crawler documentation
- BitTorrent BEP 51
- BitTorrent BEP 5
- BitTorrent BEP 33
- BitTorrent BEP 10
- Kademlia paper
- Magnetico metadata sink
- Magnetico project
- Anacrolix torrent repository
- Anacrolix GitHub repositories
- Anacrolix DHT implementation
- Anacrolix torrent notes
- Uber-go Zap logger