Metrics quickstart

This section documents how to enable metrics reporting on the Policy Server.

Before continuing, make sure you completed the previous OpenTelemetry section of this book. You need it for this section to work correctly.

You use Prometheus to collect metrics exposed by the Policy Server.

Install Prometheus

You use Prometheus Operator, that lets us define Prometheus' targets.

There are many ways to install and set up Prometheus. For ease of deployment, you should use the Prometheus Operator Helm chart from the SUSE Application Collection.

The Prometheus Operator deployed with this Helm chart uses Service Monitors, to define which services Prometheus monitors.

In your case, you are adding a ServiceMonitor targeting the suse-sec-adm-controller namespace for services that match labels kubewarden/policy-server: default and app.kubernetes.io/component: controller. This configures the Prometheus Operator to inspect which Kubernetes Endpoints belong to services matching these conditions.

You can create the two ServiceMonitors named kubewarden and kubewarden-controller for use by the default Prometheus instance installed by the Helm chart.

To do that, you should create the following values file:

cat <<EOF > kube-prometheus-stack-values.yaml
prometheus:
  additionalServiceMonitors:
    - name: kubewarden
      selector:
        matchLabels:
          kubewarden/policy-server: default
      namespaceSelector:
        matchNames:
          - suse-sec-adm-controller
      endpoints:
        - port: metrics
          interval: 10s
    - name: kubewarden-controller
      selector:
        matchLabels:
          app.kubernetes.io/component: controller
      namespaceSelector:
        matchNames:
          - suse-sec-adm-controller
      endpoints:
        - port: metrics
          interval: 10s
EOF

You should install the Prometheus Operator Helm Chart:

kubectl create namespace prometheus

kubectl create secret docker-registry application-collection \
  --namespace prometheus \
  --docker-server=dp.apps.rancher.io \
  --docker-username=<your-username> \
  --docker-password=<your-token>

helm install --wait \
  --namespace prometheus \
  --version 92.1.0 \
  --values kube-prometheus-stack-values.yaml \
  --set global.imagePullSecrets={application-collection} \
  prometheus oci://dp.apps.rancher.io/charts/prometheus-operator

Install Admission Controller

You can now install Admission Controller, in the recommended way with Helm charts.

The cert-manager is a requirement of OpenTelemetry, but you have already installed it in a previous section of this documentation.

First, create the namespace and the image pull secret for Admission Controller:

kubectl create namespace suse-sec-adm-controller

kubectl create secret docker-registry application-collection \
  --namespace suse-sec-adm-controller \
  --docker-server=dp.apps.rancher.io \
  --docker-username=<your-username> \
  --docker-password=<your-token>

Now you can deploy the rest of the Admission Controller stack. The suse-security-admission-controller Helm chart creates a PolicyServer named default.

You should configure the Helm chart so that you have metrics enabled in Admission Controller. The kubewarden-values.yaml file should have the following contents:

telemetry:
  mode: sidecar
  metrics: True
  sidecar:
    metrics:
      port: 8080

Now, install the Helm charts:

helm install --wait \
  --namespace suse-sec-adm-controller \
  --values kubewarden-values.yaml \
  --set global.imagePullSecrets={application-collection} \
  --set recommendedPolicies.enabled=True \
  --set recommendedPolicies.defaultPolicyMode=monitor \
  suse-sec-adm-controller oci://dp.apps.rancher.io/charts/suse-security-admission-controller

This creates the default instance of PolicyServer:

kubectl get policyservers.policies.kubewarden.io
NAME      AGE
default   3m7s

By default, this policy server doesn’t have metrics enabled.

Accessing Prometheus

Prometheus exposes a UI that you can use to inspect metrics exposed by different components within your Kubernetes cluster.

You can forward the Prometheus port so you can access it.

kubectl port-forward -n prometheus --address 0.0.0.0 svc/prometheus-operated 9090

Now, you can visit Prometheus on port 9090 and perform a query, for example: kubewarden_policy_evaluations_total. You can see that the number of evaluations grows over time as more requests go through the policy.

Accessing Grafana

You can forward the Grafana service so you can access it.

kubectl port-forward -n prometheus --address 0.0.0.0 svc/prometheus-grafana 8080:80

You can now login with the username admin. The Helm chart generates a random password for this user. Retrieve it with:

kubectl get secret -n prometheus prometheus-grafana \
  -o jsonpath="{.data.admin-password}" | base64 --decode; echo

Using the Admission Controller Grafana dashboard

Admission Controller has a Grafana dashboard with basic metrics giving an overview about how Admission Controller behaves in the cluster. This dashboard is available in the GitHub releases of the Admission Controller policy-server repository as a JSON file or at the Grafana website.

To import the dashboard into your environment, you can download the JSON file from the Grafana website or the repository:

curl https://github.com/kubewarden/policy-server/releases/latest/download/kubewarden-dashboard.json

Once you have the file you should access the Grafana dashboard and import it. Visit /dashboard/import in the Grafana dashboard and follow these steps:

  1. Copy and paste the JSON file contents into the Import via panel json box in the Grafana UI.

  2. Click the Load button.

  3. Choose Prometheus as the source.

  4. Click the Import button.

Another option is import it directly from the grafana.com website. For this:

  1. Copy the dashboard ID from the dashboard page,

  2. Paste it into the Import via grafana.com field

  3. Click the load button.

  4. After importing the dashboard, define the Prometheus data source to use and finish the import process.

The Grafana dashboard has panes showing the state of all policies managed by Admission Controller. It also has policy-specific panels.

You can obtain detailed metrics for a specific policy by changing the value of the policy_name variable to match the required policy’s name.

You should be able to see a dashboard similar to this:

Dashboard