At KubeCon EU 2016 in London, I gave a first talk about using BPF and Kubernetes together. I was presenting a proof of concept to introduce various degraded network scenarios in specific pods for testing the reliability of apps. There was not a lot of BPF + Kubernetes talks back then. In the meantime, Kinvolk has worked on various projects mixing Kubernetes and BPF together. The latest such project is our own Inspektor Gadget, a collection of “gadgets” for debugging and inspecting Kubernetes applications.
Today I would like to introduce Inspektor Gadget’s newest gadget that helps to write proper Kubernetes network policies.
Writing Kubernetes network policies easily
Securing your Kubernetes clusters is a task that involves many aspects: controlling what goes into your container images, writing RBAC rules for different users and services, etc. Here I focus on one important aspect: network policies.
At Kinvolk we regularly do security assessments of Kubernetes in the form of penetration testing for customers. Sometimes, the application is Kubernetes native and the network policies are developed at the same time as the application. This is ideal because the development team has a clear idea of which pod is supposed to talk to which pod. But sometimes, a pre-Kubernetes application is ported to Kubernetes and the developer tasked with writing the network policies may not have a clear idea of the architecture. Architecture documents might be missing or incomplete. Adding pod security as an afterthought might not be ideal, but thankfully the Network Policy Advisor in Inspektor Gadget can help us here.
The Network Policy Advisor workflow
A workflow we suggest that can improve things is to deploy the application in a development cluster and let Inspektor Gadget monitor and analyse the network traffic so it can suggest network policies. The developer can then review the output and add them in the project.
We will use GoogleCloudPlatform’s microservices-demo application as an example. Its kubernetes-manifests.yaml contains various deployments and services but no network policies.
After preparing a “demo” namespace, let’s ask Inspektor Gadget to monitor the network traffic from this namespace:
$ kubectl gadget network-policy monitor \ --namespaces demo \ --output ./networktrace.log
While it’s running in the background, deploy the application in the demo namespace from another terminal:
$ wget -O network-policy-demo.yaml https://raw.githubusercontent.com/GoogleCloudPlatform/microservices-demo/ccff406cdcd3e043b432fe99b4038d1b4699c702/release/kubernetes-manifests.yaml $ kubectl apply -f network-policy-demo.yaml -n demo
Once the demo is deployed and running correctly, we can see all the pods in the demo namespace:
$ kubectl get pod -n demo NAME READY STATUS RESTARTS AGE adservice-58c85c77d8-k5667 1/1 Running 0 44s cartservice-579bdd6865-2wcbk 0/1 Running 1 45s checkoutservice-66d68cbdd-smp6w 1/1 Running 0 46s currencyservice-65dd85f486-62vld 1/1 Running 0 45s emailservice-84c98657cb-lqwfz 0/1 Running 2 46s frontend-788f7bdc86-q56rw 0/1 Running 1 46s loadgenerator-7699dc7d4b-j6vq6 1/1 Running 1 45s paymentservice-5c54c9887b-prz7n 1/1 Running 0 45s productcatalogservice-7df777f796-29lmz 1/1 Running 0 45s recommendationservice-89547cff8-xf4mv 0/1 Running 1 46s redis-cart-5f59546cdd-6rq8f 0/1 Running 2 44s shippingservice-778db496dd-mhdk5 1/1 Running 0 45s
At this point, the different pods will have communicated with each other. The networktrace.log file contains one line per TCP connection with enough details to be able to infer network policies later on.
Let’s stop the network monitoring by Inspektor Gadget using Ctrl-C, and generate the Kubernetes network policies:
$ kubectl gadget network-policy report \ --input ./networktrace.log > network-policy.yaml
Note: Here we are running Inspektor Gadget as a kubectl subcommand. You could
also run it as a stand-alone binary using
One of the network policies it creates is for the cartservice: Inspektor Gadget noticed that it received connections from the frontend and initiated connections to redis-cart. It displays the following suggestion accordingly:
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: creationTimestamp: null name: cartservice-network namespace: demo spec: egress: - ports: - port: 6379 protocol: TCP to: - podSelector: matchLabels: app: redis-cart ingress: - from: - podSelector: matchLabels: app: frontend ports: - port: 7070 protocol: TCP podSelector: matchLabels: app: cartservice policyTypes: - Ingress - Egress
As you can see, it converted the set of connection tuples into a set of network policies using usual Kubernetes label selectors instead of IP addresses.
Of course, those automatically-produced network policies should not be used blindly: a developer should verify that the connections observed are legitimate. The Network Policy Advisor gadget has some limitations too (see #39), but it’s a lot easier to review them and possibly make some small changes, rather than writing them from scratch with a frustrating trial and error development cycle. This saves precious development time and likely costs too.
Inspektor Gadget has useful features for developers of Kubernetes applications. As an Open Source project, contributions are welcome. Join the discussions on the #inspektor-gadget channel in the Kubernetes Slack or, if you want to know about our services related to pentesting and security, reach us at [email protected].