Custom OpenTelemetry Collector

This guide explains how to configure Admission Controller to send telemetry data to an OpenTelemetry collector already deployed on the cluster.

You should deploy only one instance of the OpenTelemetry Collector in the cluster.

Install dependencies

First, begin by installing the dependencies of OpenTelemetry Collector.

You need encrypted communication between the SUSE Security Admission Controller components and the collector. You can use cert-manager to manage all the certificates required for secure communication.

OpenTelemetry Collector traces get sent to a Jaeger instance.

The Admission Controller stack sends metrics to the OpenTelemetry Collector. This one exposes the metrics as a Prometheus endpoint. The metrics are then collected by a Prometheus instance and stored in its database. The same Prometheus instance also exposes a UI to view and use the metrics.

All the Helm charts and container images used in this guide come from the SUSE Application Collection. Log in to the registry and create the image pull secrets as described in the OpenTelemetry quick start.

Resources you create get defined in the suse-sec-adm-controller Namespace, or expect its existence. Due to that, you should begin by creating the Namespace and its image pull secret:

kubectl create namespace suse-sec-adm-controller

kubectl create secret docker-registry application-collection \
  --namespace suse-sec-adm-controller \
  --docker-server=dp.apps.rancher.io \
  --docker-username=<your-username> \
  --docker-password=<your-token>

Install cert-manager and OpenTelemetry

You install cert-manager and OpenTelemetry operator in this way:

for ns in cert-manager open-telemetry; do
  kubectl create namespace $ns
  kubectl create secret docker-registry application-collection \
    --namespace $ns \
    --docker-server=dp.apps.rancher.io \
    --docker-username=<your-username> \
    --docker-password=<your-token>
done

helm install --wait \
  --namespace cert-manager \
  --set crds.enabled=true \
  --set global.imagePullSecrets={application-collection} \
  --version 1.21.2 \
  cert-manager oci://dp.apps.rancher.io/charts/cert-manager

helm install --wait \
  --namespace open-telemetry \
  --version 0.124.1 \
  --set global.imagePullSecrets={application-collection} \
  --set manager.collectorImage.tag=0.160.0-contrib \
  my-opentelemetry-operator oci://dp.apps.rancher.io/charts/opentelemetry-operator

You setup communication between Admission Controller components and the OpenTelemetry Collector using mTLS.

To do that, you need to create the whole Public Key Infrastructure (PKI):

# pki.yaml file
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-client-certificate
  namespace: suse-sec-adm-controller
spec:
  dnsNames:
  - kubewarden.suse-sec-adm-controller.svc
  - kubewarden.suse-sec-adm-controller.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: my-selfsigned-issuer
  secretName: my-client-cert
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-certificate
  namespace: suse-sec-adm-controller
spec:
  dnsNames:
  - my-collector-collector.suse-sec-adm-controller.svc
  - my-collector-collector.suse-sec-adm-controller.svc.cluster.local
  issuerRef:
    kind: Issuer
    name: my-selfsigned-issuer
  secretName: my-server-cert
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: my-selfsigned-issuer
  namespace: suse-sec-adm-controller
spec:
  selfSigned: {}

Apply the manifest:

kubectl apply -f pki.yaml

Install Jaeger and Prometheus

After that, you install Jaeger to store and visualize trace events.

kubectl create namespace jaeger

kubectl create secret docker-registry application-collection \
  --namespace jaeger \
  --docker-server=dp.apps.rancher.io \
  --docker-username=<your-username> \
  --docker-password=<your-token>

helm upgrade -i --wait \
  --namespace jaeger \
  --version 2.57.0 \
  jaeger-operator oci://dp.apps.rancher.io/charts/jaeger-operator \
  --set global.imagePullSecrets={application-collection} \
  --set rbac.clusterRole=true

kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: my-open-telemetry
  namespace: jaeger
spec:
  imagePullSecrets:
    - name: application-collection
EOF

If the Jaeger Operator webhook isn’t ready yet, creating the Jaeger resource fails with failed calling webhook "mjaeger.kb.io". Wait a few seconds and run the command again.

If you installed Traefik in the OpenTelemetry quickstart, expose the Jaeger Query UI with this Ingress:

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-open-telemetry-query
  namespace: jaeger
spec:
  ingressClassName: traefik
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: my-open-telemetry-query
                port:
                  number: 16686
EOF

Now you install Prometheus to store and visualize metrics.

cat <<EOF > kube-prometheus-stack-values.yaml
prometheus:
  additionalServiceMonitors:
  - name: kubewarden
    selector:
      matchLabels:
        app.kubernetes.io/instance: suse-sec-adm-controller.my-collector
    namespaceSelector:
      matchNames:
        - suse-sec-adm-controller
    endpoints:
    - port: prometheus
      interval: 10s
EOF

kubectl create namespace prometheus

kubectl create secret docker-registry application-collection \
  --namespace prometheus \
  --docker-server=dp.apps.rancher.io \
  --docker-username=<your-username> \
  --docker-password=<your-token>

helm install --wait \
  --namespace prometheus \
  --version 92.1.0 \
  --values kube-prometheus-stack-values.yaml \
  --set global.imagePullSecrets={application-collection} \
  prometheus oci://dp.apps.rancher.io/charts/prometheus-operator

The Prometheus service monitor obtains the Admission Controller metrics by scraping the OpenTelemetry collector running in the suse-sec-adm-controller Namespace.

Install OpenTelemetry Collector

Now you can deploy a custom OpenTelemetry Collector in the suse-sec-adm-controller Namespace. The collector uses a ServiceAccount with the image pull secret, so it can pull its container image from the SUSE Application Collection.

# otel-collector.yaml file
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-collector
  namespace: suse-sec-adm-controller
imagePullSecrets:
  - name: application-collection
---
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: my-collector
  namespace: suse-sec-adm-controller
spec:
  mode: deployment # This configuration is omittable.
  serviceAccount: my-collector
  volumes:
  - name: server-certificate
    secret:
      secretName: my-server-cert
  - name: client-certificate
    secret:
      secretName: my-client-cert
  volumeMounts:
  - name: server-certificate
    mountPath: /tmp/etc/ssl/certs/my-server-cert
    readOnly: true
  - name: client-certificate
    mountPath: /tmp/etc/ssl/certs/my-client-cert
    readOnly: true
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            tls:
              cert_file: /tmp/etc/ssl/certs/my-server-cert/tls.crt
              key_file: /tmp/etc/ssl/certs/my-server-cert/tls.key
              client_ca_file: /tmp/etc/ssl/certs/my-client-cert/ca.crt
    processors: {}
    exporters:
      debug:
        verbosity: normal
      prometheus:
        endpoint: ":8080"
      otlp/jaeger:
        endpoint: "my-open-telemetry-collector.jaeger.svc.cluster.local:4317"
        tls:
          insecure: true
    service:
      pipelines:
        metrics:
          receivers: [otlp]
          processors: []
          exporters: [debug, prometheus]
        traces:
          receivers: [otlp]
          processors: []
          exporters: [debug, otlp/jaeger]

Apply the manifest:

kubectl apply -f otel-collector.yaml

That configuration uses a trivial processing pipeline to receive trace events and to forward them to Jaeger. It also receives metrics and exposes them for collection by Prometheus.

You secure communication between the Admission Controller stack and the OpenTelemetry Collector using mTLS. However the communication between the OpenTelemetry Collector and Jaeger isn’t secured, to reduce the complexity of the example.

Install Admission Controller stack

When the OpenTelemetry Collector is running, you can deploy Admission Controller in the usual way.

You need to configure the Admission Controller components so they send events and metrics to the OpenTelemetry Collector.

# values.yaml
telemetry:
  mode: custom
  metrics: True
  tracing: True
  custom:
    endpoint: "https://my-collector-collector.suse-sec-adm-controller.svc:4317"
    insecure: false
    otelCollectorCertificateSecret: "my-server-cert"
    otelCollectorClientCertificateSecret: "my-client-cert"

The Secret referenced by the otelCollectorCertificateSecret key must have an entry named ca.crt. That holds the certificate of the CA that issued the certificate used by the OpenTelemetry Collector.

The Secret referenced by the otelCollectorClientCertificateSecret key must have the following entries: tls.crt and tls.key keys. These are the client certificate and its key used by the Admission Controller stack to authenticate against the OpenTelemetry Collector.

Leave these values empty if you don’t use encryption or mTLS.

Install the Admission Controller stack:

helm install --wait \
  --namespace suse-sec-adm-controller \
  --values values.yaml \
  --set global.imagePullSecrets={application-collection} \
  --set recommendedPolicies.enabled=True \
  --set recommendedPolicies.defaultPolicyMode=monitor \
  suse-sec-adm-controller oci://dp.apps.rancher.io/charts/suse-security-admission-controller

Now everything is in place.

Exploring the Jaeger UI

You can see the trace events generated by Admission Controller by using the Jaeger web UI. They’re grouped under the kubewarden-policy-server service:

The Jaeger dashboard

To access the Jaeger UI with Traefik, port-forward the Traefik Service:

kubectl -n traefik port-forward service/traefik 8080:80

The web UI is reachable at http://localhost:8080.

If you prefer to bypass Traefik, port-forward the Jaeger Query service directly:

kubectl -n jaeger port-forward service/my-open-telemetry-query 16686

The web UI is reachable at http://localhost:16686.

Exploring the Prometheus UI

You can access the Prometheus UI by port forwarding to your local machine:

kubectl port-forward -n prometheus --address 0.0.0.0 svc/prometheus-operated 9090

The web UI is now reachable at http://localhost:9090.