Using trace oomkill

    The trace oomkill gadget is used to monitor when out-of-memory killer kills a process.

    How to use it?

    First, we need to create one pod with limited amount of memory:

    $ kubectl apply -f docs/examples/limited-memory-pod.yaml
    namespace/oomkill-demo created
    pod/test-pod created
    

    You can now use the gadget, but output will be empty:

    $ kubectl gadget trace oomkill -n oomkill-demo
    NODE             NAMESPACE        POD              CONTAINER        KPID   KCOMM            PAGES  TPID             TCOMM
    

    The gadget is waiting for the OOM killer to get triggered and kill a process in oomkill-demo namespace (alternatively, we could use -A and get out-of-memory killer events in all namespaces).

    To trigger the OOM killer, in another terminal, exec a container and run this command to exhaust the memory:

    $ kubectl get pod -n oomkill-demo
    NAME       READY   STATUS    RESTARTS   AGE
    test-pod   1/1     Running   0          52s
    $ kubectl exec -n oomkill-demo -ti test-pod -- tail /dev/zero
    command terminated with exit code 137
    

    Go back to the first terminal and see:

    NODE             NAMESPACE        POD              CONTAINER        KPID   KCOMM            PAGES  TPID             TCOMM
    minikube         oomkill-demo     test-pod         test-container   11507  tail             32768  11507  tail
    

    The printed lined corresponds to the killing of the perl process by the OOM killer. Here is the full legend of all the fields:

    • KPID: The PID of the process killed by the OOM killer (KilledPID).
    • KOMM: The command of the process killed by the OOM killer (KilledCommand).
    • PAGES: The number of pages the killed process had. A page is 4096 bytes on majority of operating system.
    • TPID: The PID of the process which triggered the OOM killer (TriggeredPID).
    • TCOMM: The command of the process which triggered the OOM killer (TriggeredCommand).

    The line shown above can also be read like this: “The tail command, with PID 11507, running inside the test-container container, in the test-pod pod, in the oomkill-demo namespace, on the minikube node, was killed by the OOM killer because it allocated 32768 pages. The OOM killer was triggered by tail with PID 11507.”

    Note that, in this case, the command which was killed by the OOM killer is the same which triggered it, this is not always the case.

    Only print some information

    You can restrict the information printed using -o custom-columns=column0,...,columnN. This command will only show the PID and command of the killed process:

    $ kubectl gadget trace oomkill -A -o custom-columns=kpid,kcomm
    KPID   KCOMM
    15182  tail
    

    The following command is the same as default printing:

    $ kubectl gadget trace oomkill -A -o custom-columns=node,namespace,container,pod,kpid,kcomm,pages,tpid,tcomm
    NODE             NAMESPACE        CONTAINER        POD              KPID   KCOMM            PAGES  TPID   TCOMM
    minikube         oomkill-demo     test-container   test-pod         17349  tail             32768  17349  tail
    

    Use JSON output

    This gadget supports JSON output, for this simply use -o json:

    $ kubectl gadget trace oomkill -A -o json
    {"type":"normal","node":"minikube","namespace":"oomkill-demo","pod":"test-pod","container":"test-container","tpid":13416,"tcomm":"tail","kpid":13416,"kcomm":"tail","pages":32768,"mountnsid":4026532588}
    # You can use jq to make the output easier to read:
    $ kubectl gadget trace oomkill -A -o json | jq
    {
      "type": "normal",
      "node": "minikube",
      "namespace": "oomkill-demo",
      "pod": "test-pod",
      "container": "test-container",
      "tpid": 13416,
      "tcomm": "tail",
      "kpid": 13416,
      "kcomm": "tail",
      "pages": 32768,
      "mountnsid": 4026532588
    }
    

    Clean everything

    Congratulations! You reached the end of this guide! You can now delete the resource we created:

    $ kubectl delete -f docs/examples/limited-memory-pod.yaml
    namespace "oomkill-demo" deleted
    pod "test-pod" deleted