SUSE Observability Cluster performance test

Overview

This page provides a method to profile a running cluster to verify it is performing according to expectation. This can be used when experiencing degraded performance to actively investigate how several parts are performing. This is an addition to the Support Package (Logs) collector for collecting logs in general.

Notes

The script should be executed from a host that has kubectl configured with access to the SUSE® Observability cluster or from any other host with the necessary permissions and connectivity to the cluster. This can be done directly on the host using the root user or via sudo. If kubeconfig is not set, use the command export KUBECONFIG=$PATH-TO-YOUR/kubeconfig.

Usage

The script needs to be downloaded and run directly on the host, using the root user or sudo.

Download and run the script

Download the suse-observability_performance_collector.sh script.

Save the script as: suse-observability_performance_collector.sh
Run the script using the following commands:
```
bash suse-observability_performance_collector.sh
```
This will generate a file in the current directory named suse-observability_performance_<date>.tar.gz. Upload this file to the case.

Additional arguments can be specified. For example, to choose a different namespace:

> bash suse-observability_performance_collector.sh -h
SUSE Observability performance measurement tool.
Runs some rudimentary performance tests on a deployed instance to validate performance.

Usage: $0 [options] [<namespace>]

options:
  -h  Print this help

<namespace>:
  The namespace that is running SUSE Observability, or
  "suse-observability" when not specified

Reference performance

Below is an output we use as reference for the performance of a system, which we also use to tweak our own profiles. Customer disk and speeds should approach these numbers.

=== SUSE Observability Performance Summary ===
Date: 2026-03-09T09:46:18Z

--- Hdfs Disk Buffered ---
  suse-observability-hbase-hdfs-dn-0                           151 MB/s
  suse-observability-hbase-hdfs-dn-1                           150 MB/s
  suse-observability-hbase-hdfs-dn-2                           150 MB/s

--- Hdfs Disk Direct ---
  suse-observability-hbase-hdfs-dn-0                           58.3 MB/s
  suse-observability-hbase-hdfs-dn-1                           56.6 MB/s
  suse-observability-hbase-hdfs-dn-2                           56.7 MB/s

--- Kafka Disk Buffered ---
  suse-observability-kafka-0                                   173 MB/s
  suse-observability-kafka-1                                   142 MB/s
  suse-observability-kafka-2                                   143 MB/s

--- Kafka Disk Direct ---
  suse-observability-kafka-0                                   59.2 MB/s
  suse-observability-kafka-1                                   59.2 MB/s
  suse-observability-kafka-2                                   59.4 MB/s

--- Kafka Producer Local ---
  suse-observability-kafka-0                                   50241.157556 records/sec (49.06 MB/sec), 557.44 ms avg latency
  suse-observability-kafka-1                                   31422.825540 records/sec (30.69 MB/sec), 903.49 ms avg latency
  suse-observability-kafka-2                                   31703.760066 records/sec (30.96 MB/sec), 893.00 ms avg latency

--- Kafka Producer Remote ---
  suse-observability-kafka-0                                   59765.718384 records/sec (58.36 MB/sec), 453.19 ms avg latency
  suse-observability-kafka-1                                   54656.755575 records/sec (53.38 MB/sec), 500.53 ms avg latency
  suse-observability-kafka-2                                   39503.831872 records/sec (38.58 MB/sec), 703.39 ms avg latency

--- Hdfs Network ---
  suse-observability-hbase-hdfs-dn-0 -> suse-observability-hbase-hdfs-dn-1 571 MB/s
  suse-observability-hbase-hdfs-dn-1 -> suse-observability-hbase-hdfs-dn-2 524 MB/s
  suse-observability-hbase-hdfs-dn-2 -> suse-observability-hbase-hdfs-dn-0 597 MB/s

Clarifications:

Buffered IO relates to raw disk throughput, allowing data to buffer in the kernel
Direct disk measures disk throughput using O_DIRECT, disallowing buffering, which more closely approximates our databases using fsync() and gives insight in the latency of the underlying storage
Kafka Producer local puts data into kafka over localhost, remote over the network. Remote can be faster than local when CPU is the bottleneck rather than network (which was the case here, causing remote to be more efficient).