Prometheus on Kubernetes

Deploying Prometheus

# Create the namespace and CRDs, and then wait for them to be available before creating the remaining resources
kubectl apply --server-side -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl apply -f manifests/

Configuring Ingress for Grafana (and Optionally Grafana)

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-service-ingress
  namespace: monitoring
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
  - host: grafana.k8stest.jmoore53.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: grafana
            port:
              number: 3000
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-service-ingress
  namespace: monitoring
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
  - host: prometheus.k8stest.jmoore53.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-k8s
            port:
              number: 9090

Configuring Monitoring for Traefik!

Note this all worked because traefik and its metrics were in the default namespace.

Service/Traefik-metrics.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-metrics
  labels:
    app: traefik-metrics
spec:
  type: ClusterIP
  selector:
    app: traefik
  ports:
    - name: metrics
      protocol: TCP
      port: 8080
      # The targetPort needs to be configured for the port on the pod to forward traffic to
      # https://stackoverflow.com/a/61452441
      # Originally this was set as such
      #   targetPort: metrics
      # Which was incorrect.. The Service and ServiceMonitor were confused.
      targetPort: 8080

ServiceMonitor/TraefikMonitoring.yaml

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: monitoring
  name: traefik-metrics
  labels:
    app: traefik-metrics
    release: prometheus-operator
spec:
  # Search the application based on the app label
  selector:
    matchLabels:
      app: traefik-metrics
  # Match the namespace..
  namespaceSelector:
    matchNames:
    - default
    #any: true
  endpoints:
  - path: /metrics
    # This is named in the Service itself
    port: metrics

Other Namespaces and a look at ArgoCD and why I couldn’t get it to work

Kube-Prometheus was built for you guessed it, prometheus monitoring for kubernetes. This means other namespaces are not enabled by default.

The way I fixed this was by updating the kube-prometheus/manifests/prometheus-clusterRole.yaml to an insecure format allowing kube-prometheus to monitor all namespaces instead of just kube-system and default.

The configuration of prometheus-clusterRole.yaml now looks like the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.34.0
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
- nonResourceURLs:
  - /metrics
  verbs:
  - get

After re-applying the configuration with the following, everything was good to go:

kubectl apply -f manifests/

This was definitely a lesson for me in permissions.

Configuring Alerts for Argocd

After these are configured properly, grafana should properly show clusters, applications, repositories, and operators.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: monitoring
  name: argocd-metrics
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app: argocd-metrics
  namespaceSelector:
    matchNames:
    - argocd
  endpoints:
  - port: metrics
    path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: monitoring
  name: argocd-server-metrics
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app: argocd-server-metrics
  namespaceSelector:
    matchNames:
    - argocd
  endpoints:
  - port: metrics
    path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  namespace: monitoring
  name: argocd-repo-server-metrics
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app: argocd-repo-server
  namespaceSelector:
    matchNames:
    - argocd
  endpoints:
  - port: metrics
    path: /metrics

Space?

“Delete the pods” ?

Opening Grafana Console

# From within the control pane to expose the service
k port-forward svc/grafana -n monitoring --address 0.0.0.0 3000

Username/admin Password/admin

Jack Moore

Email: jack(at)jmoore53.com