Jack Moore

Email: jack(at)jmoore53.com
Project Updates

Ingress, Allowing Outside Requests In

02 Feb 2022 » kubernetes, containers, networking, ingress

Requests need a way to get in. Initially this is where I was using MetalLB to solve my problem. I would create a service using the MetalLB LoadBalancer configuration specifying an IP address, and the service would map 1:1 to a pod where the application was running. Now I am using an ingress to point multiple applications to the same DNS entry with paths. This means any request to k8s.jmoore53.com now gets routed to the ingress, which then points to the service, which then points to the pod.

An extra hop is being added, but the configuration for this is all done with DNS now.

Starting with What I Know, NGINX

Because I am already familiar with NGINX and the way it is configured in a standard installation, I thought it would be the best solution out of the box. The installation looked like the following (of course I pulled down the deploy.yaml):

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
kubectl apply -f deploy.yaml

For this installation, I actually made no modifications to the deploy.yaml before deploying it.

After it was installed, I needed to create the ingress/service/pod yaml example which looked like the following:

# Ingress!
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: "k8stest.jmoore53.com"
    http:
      paths:
      - path: "/testpath"
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
      - path: "/"
        pathType: Prefix
        backend:
          service:
            name: nginx-service
            port:
              number: 80
---
# Service
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx
---
# Pod Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

After this was created, the Ingress was assigned an IP and requests to “/” and “/testpath” were routing properly to the nginx service which routed to the nginx pod. This is all fine and great, but was a bit difficult to troubleshoot at times because everything was configured with NGINX.. Aka, any kind of nginx error was a bit difficult to troubleshoot because the ingress and the pods showed the same 404 error (oops, however this can be solved with a whoami pod instead of an nginx pod).

Moving To Traefik!

Nginx was a great solution, but Traefik is another solution available for Ingress, so I wanted to check it out. (It also provides support for API Gateway (L7/L4 support), and has a pretty cool dashboard for all the routes.)

As a quick install, I used the “Configuring Kubernetes Ingress Controller” from the Traefik documentation. I created the rbac.yaml, ingress.yaml, traefik.yaml, and whoami.yaml files.

In a short and somewhat unhelpful description, the files do the following:

  • rbac.yaml: Installs the cluster roles and bindings
  • traefik.yaml: configures a ServiceAccount, a traefik Deployment, and a Service, see below, also note there is a container port exposing 8080 for the admin port
  • whoami.yaml: A simple whoami service and deployment, this is a really basic web page that deploys information on a get request basically
  • ingress.yaml: This is similar to the nginx ingress.
# traefik.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: traefik
  labels:
    app: traefik
spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.6
          args:
            - --api
            - --api.insecure=true
            - --api.dashboard=true
            - --entrypoints.web.address=:80
            - --providers.kubernetesingress
          ports:
            - name: web
              containerPort: 80
            - name: admin
              containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: traefik
spec:
  type: LoadBalancer
  selector:
    app: traefik
  ports:
    - protocol: TCP
      port: 80
      name: web
      targetPort: 80
    - protocol: TCP
      port: 8080
      name: admin
      targetPort: 8080

Note this is similar, if not for a few lines almost identical to the NGINX ingress.

# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myingress
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
    - host: k8stest.jmoore53.com
      http:
        paths:
          - path: /bar
            pathType: Exact
            backend:
              service:
                name:  whoami
                port:
                  number: 80
          - path: /foo
            pathType: Exact
            backend:
              service:
                name: whoami
                port:
                  number: 80

Gateway API with Traefik

Since the Gateway API is in Alpha, it needs to be installed with the following:

kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v0.4.0" \
| kubectl apply -f -

Then from here I used the Traefik Kubernetes Gateway Provider.

Essentially it’s similar to the traffic ingress. Below are the files:

  • rbac.yaml: Access Control
  • whoami.yaml: Again, whoami service returning request
  • traefik-service.yaml: Service account, Deployment, and Service for Traefik
  • gateway-api.yaml: Has a GatewayClass, a Gateway, and an HTTPRoute which looks like the following:
# gateway-api.yaml
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GatewayClass
metadata:
  name: my-gateway-class

spec:
  controllerName: traefik.io/gateway-controller

---
kind: Gateway
metadata:
  name: my-gateway

spec:
  gatewayClassName: my-gateway-class
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      #tls:
      #  certificateRefs:
      #    - kind: Secret
      #      name: mysecret

---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
  name: http-app
  namespace: default

spec:
  parentRefs:
    - name: my-gateway

  hostnames:
    - apigw.k8stest.jmoore53.com

  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /foo

        - path:
            type: PathPrefix
            value: /bar

      backendRefs:
        - name: whoami
          port: 80
          weight: 1

I did end up disabling all traffic over 443 and SSL for testing purposes. Also note, basically all of these are default configurations.

SSL?

Right now SSL Termination is done on the HAProxy end and then data is proxied back to the Kubernetes service.

© Jack Moore