Jack Moore

Email: jack(at)jmoore53.com
Project Updates

BGP and Cilium Kubernetes

20 Sep 2022 » k8s, networking, firewall

For the current setup I have with kubernetes I ended up using Cilium as the networking provider configured with BGP on the OPNSense Firewall I have. This setup was a bit different because Cilium didn’t support having Nodes on the same network as the pods.. which makes sense, however it should have allowed for better configuration of the pod CIDR. Below is how I configured Cilium and OPNSense:

I made an attempt to run clium configured with BGP, but needed to move the nodes to network.

This was the process I used:

  • Create new vlan on switch for network;
    • Provisioning of nodes was done with Terraform;
  • Pluck off a space in the network for kubernetes;
  • Add Firewall interface for the new L2 network
  • Update Terraform to move everything to a network;
  • Update Scripts
  • Try Cilium again;

Installing Helm and Cilium

curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium --version 1.12.2   --namespace kube-system   --set bgp.enabled=true   --set bgp.announce.loadbalancerIP=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true #--set bgp.announce.podCIDR=true

After installing Clium I needed to add a config map for bgp pointing to opnsense (note this may have needed to have been done before installing cilium):

  • Note the peer address is my firewall ip
  • Note the peer asn is the AS of the firewall
  • Note the current AS is the AS of the kubernetes cluster
apiVersion: v1
kind: ConfigMap
  name: bgp-config
  namespace: kube-system
  config.yaml: |
      - peer-address:
        peer-asn: 64512
        my-asn: 64513
      - name: default
        protocol: bgp

OPNSense Side:

On the opnsense side I had to install the frr package which allowed for better routing. From there I had to configure BGP.

This meant:

  • enabling bgp
  • Setting the BGP AS Number to 64512
  • Adding all the k8s nodes into the neigbors of the bgp config with the correct peer asn (peer asns are 64513)

After everything has been configured LoadBalancer IPs can now be used on the network.

© Jack Moore