Debugging Kubernetes Webhooks Locally (via ngrok)

We recently had the need to write a Mutating Admissions Webhook in k8s to monitor and mutate jobs being submitted to the cluster. The reason why is for another post – but the first thing I wanted to do was get to the point where I could hit a breakpoint and see what was going on.

This was trickier than I first thought, given the certificates involved and the way that webhooks run in the cluster. We used the very helpful Kubebuilder project to scaffold out our webhooks, but there’s no straightforward way of running one locally to see what k8s passes to it.

My colleague Bart Jansen and I eventually found a way to ‘attach’ a locally running webhook to a running cluster to step through our webhook code, via ngrok. Think remote debug.

Anatomy

Usually the webhook runs like so:
[K8s fires webhook event] –> [Service] –> [Endpoint] –> [pod running webhook code].
We can see this if we look at the service object deployed via kubebuilder:

NAMESPACE NAME ENDPOINTS
…
psc-system psc-webhook-service 10.244.0.9:9443
…

Here kubernetes will fire the event at a service named psc-webhook-service which will use the endpoint 10.244.0.9:9443 which is the IP and port of the pod running the webhook.

Everything in a huge gif:

If you wanna sit for a minute and see a demo – take a look here:
Big demo gif 🙂

The Plan:

What we’ll do is:
– Deploy the webhook code and config (not covered here)
– Copy the cluster certs to our dev pc
– Start an ngrok session on our dev pc
– Use the ngrok IP / port as the endpoint for the service.

0) Create debug-* versions of the endpoint and service file

We will essentially be overwriting the standard service / endpoint for our running webhook to point at ngrok instead. You’ll need the following 2 files as well as some new make commands: (you’ll need to update the namespace and names appropriately)

debug-endpoint.yaml

	apiVersion: v1
	kind: Endpoints
	metadata:
	name: psc-webhook-service
	namespace: psc-system
	subsets:
	– addresses:
	– ip: 18.185.254.87 #0.tcp.eu.ngrok.io
	ports:
	– port:

view raw debug-endpoint.yaml hosted with ❤ by GitHub

debug-service.yaml

	apiVersion: v1
	kind: Service
	metadata:
	name: psc-webhook-service
	namespace: psc-system
	selfLink: /api/v1/namespaces/psc-system/services/psc-webhook-service
	spec:
	clusterIP: 10.98.155.51
	ports:
	– port: 443
	protocol: TCP
	targetPort: 9443
	sessionAffinity: None
	type: ClusterIP
	status:
	loadBalancer: {}

view raw debug-service.yaml hosted with ❤ by GitHub

Makefile

	copy-running-certs:
	kubectl get secret webhook-server-cert -n psc-system -o json | sed 's/ca.crt/cacrt/; s/tls.crt/tlscrt/; s/tls.key/tlskey/' > secrets.json
	mkdir -p ./.running-keys
	jq -r .data.cacrt < secrets.json | base64 –decode > ./.running-keys/ca.crt
	jq -r .data.tlscrt < secrets.json | base64 –decode > ./.running-keys/tls.crt
	jq -r .data.tlskey < secrets.json | base64 –decode > ./.running-keys/tls.key
	mkdir -p /tmp/k8s-webhook-server/serving-certs
	cp ./.running-keys/* /tmp/k8s-webhook-server/serving-certs/
	rm secrets.json
	create-local-certs:
	mkdir -p .keys && openssl req -nodes -new -x509 -keyout ./.keys/ca.key -out ./.keys/ca.crt -subj "/CN=cronprimer CA"
	openssl genrsa -out ./.keys/tls.key 2048
	openssl req -new -key ./.keys/tls.key -subj "/CN=webhook-server.webhook.svc" | openssl x509 -req -CA ./.keys/ca.crt -CAkey ./.keys/ca.key -CAcreateserial -out ./.keys/tls.crt
	mkdir -p /tmp/k8s-webhook-server/serving-certs
	cp ./.keys/* /tmp/k8s-webhook-server/serving-certs/
	# Deploy and configure ngrok debugger
	ngrok-debug: SHELL:=/bin/bash
	ngrok-debug:
	ngrok tcp 9443 –region eu > /dev/null & \
	sleep 3; \
	kubectl set selector service/psc-webhook-service -n psc-system ""; \
	sed -i "/^\([[:space:]]*- port: \).*/s//\1$$(curl -s localhost:4040/api/tunnels | jq -r '.tunnels[0].public_url | split(":")[-1]')/" debug/debug-endpoint.yaml; \
	kubectl apply -f debug/debug-endpoint.yaml

view raw Makefile-local-debug hosted with ❤ by GitHub

1) Copy cluster certs

To debug our code locally we need the same certs running as are running in the cluster. This way kubernetes can authenticate to our service:
make copy-running-certs

Basically this grabs the certs running in the cluster and does some pokery to base64 decode them and save them locally where our code should be looking for them.

2) Start ngrok and send your webhook events there

• Go to https://www.ngrok.com/ and sign in / sign up for free
• Go to https://dashboard.ngrok.com/auth to get a copy of your auth token and run ngrok authtoken <token>
• run make ngrok-debug to open an ngrok proxy, remove the webhook node label and apply the debug endpoint

The above ngrok-debug command (amazing work Bart!) actually updates the 2 yaml files and applies them to your cluster. Now webhook events should be being fired at your ngrok IP, and your PC should be connected to ngrok, ready to receive them.

3) F5

You should now be able to set a breakpoint in your webhook and hit F5 (or the equivalent) to run it locally. Apply something to your cluster which should result in an event being raised – in our case we were applying some test yaml which contained some dummy jobs.

That’s it… I know this is not a full step-by-step, but doing so is tricky given there are different ways to write and deploy webhooks. I hope this helps get you on the right path, anyway… 🙂