Skip to main content

Example support bundle specs

This topic includes example support bundle specifications. For more, see the Troubleshoot example repository in GitHub.

Each example is a self-contained spec. You can copy the collectors and analyzers from an example into your existing spec, or add the example to your release as its own Secret. When a support bundle is generated from a Replicated installer's Admin Console or with kubectl support-bundle --load-cluster-specs, all of the labeled specs in the cluster are combined into a single bundle. For more information, see Combine support bundle specs.

Check application health​

This example shows how to use the deploymentStatus analyzer to check an application's health. The deploymentStatus analyzer uses data from the default clusterResources collector to check if liveness probes are passing and if the required number of replicas are ready for an application deployment.

Replicated recommends using the deploymentStatus analyzer to check application health because it doesn't require any additional collectors besides clusterResources. It also requires no additional RBAC permissions. The http, exec, and runPod collectors can probe health endpoints directly, but each has environment-specific limitations:

  • http cannot resolve in-cluster DNS from the CLI.
  • exec requires pods/exec RBAC and tools such as curl, wget, or nc in the container.
  • runPod requires pods/create RBAC. In air-gapped environments, the image must already be present on the node.

For more information, see Deployment Status and Cluster Resources in the Troubleshoot documentation.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData: 
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
      collectors: []
      analyzers: 
        - deploymentStatus:
            name: api
            namespace: default
            outcomes:
              - fail:
                  when: "< 1"
                  message: The api deployment does not have any ready replicas. Check that the application's liveness probes are passing and that the application is not in a crash loop.
              - warn:
                  when: "= 1"
                  message: The api deployment has only a single ready replica.
              - pass:
                  message: The api deployment is healthy and has multiple ready replicas.

Check HTTP requests​

If your application has its own API that serves status, metrics, performance data, and so on, this information can be collected and analyzed.

The example below uses the http collector and the textAnalyze analyzer to check that an HTTP request to the Slack API at https://api.slack.com/methods/api.test made from where the support bundle is generated returns a successful response of "status": 200,.

For more information, see HTTP and Regular Expression in the Troubleshoot documentation.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData: 
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
      collectors:
        - http:
            collectorName: slack
            get:
              url: https://api.slack.com/methods/api.test
      analyzers:      
        - textAnalyze:
            checkName: Slack Accessible
            fileName: slack.json
            regex: '"status": 200,'
            outcomes:
              - pass:
                  when: "true"
                  message: "Can access the Slack API"
              - fail:
                  when: "false"
                  message: "Cannot access the Slack API. Check that the server can reach the internet and check [status.slack.com](https://status.slack.com)."

Check Kubernetes version​

The example below uses the clusterVersion analyzer to check the version of Kubernetes running in the cluster. The clusterVersion analyzer uses data from the default clusterInfo collector.

For more information, see Cluster Version and Cluster Info in the Troubleshoot documentation.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData: 
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
      collectors: []
      analyzers:      
        - clusterVersion:
            outcomes:
              - fail:
                  message: This application relies on kubernetes features only present in 1.16.0
                  and later.
                  uri: https://kubernetes.io
                  when: < 1.16.0
              - warn:
                  message: Your cluster is running a version of kubernetes that is out of support.
                  uri: https://kubernetes.io
                  when: < 1.24.0
              - pass:
                  message: Your cluster meets the recommended and quired versions of Kubernetes.

Check node resources​

The example below uses the nodeResources analyzer to check that the minimum requirements are met for memory, CPU cores, number of nodes, and ephemeral storage. The nodeResources analyzer uses data from the default clusterResources collector.

For more information, see Cluster Resources and Node Resources in the Troubleshoot documentation.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData: 
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
      collectors: []
      analyzers:
        - nodeResources:
            checkName: One node must have 2 GB RAM and 1 CPU Cores
            filters:
              allocatableMemory: 2Gi
              cpuCapacity: "1"
            outcomes:
              - fail:
                  when: count() < 1
                  message: Cannot find a node with sufficient memory and cpu
              - pass:
                  message: Sufficient CPU and memory is available
        - nodeResources:
            checkName: Must have at least 3 nodes in the cluster
            outcomes:
              - fail:
                  when: "count() < 3"
                  message: This application requires at least 3 nodes
              - warn:
                  when: "count() < 5"
                  message: This application recommends at last 5 nodes.
              - pass:
                  message: This cluster has enough nodes.
        - nodeResources:
            checkName: Each node must have at least 40 GB of ephemeral storage
            outcomes:
              - fail:
                  when: "min(ephemeralStorageCapacity) < 40Gi"
                  message: Noees in this cluster do not have at least 40 GB of ephemeral storage.
                  uri: https://kurl.sh/docs/install-with-kurl/system-requirements
              - warn:
                  when: "min(ephemeralStorageCapacity) < 100Gi"
                  message: Nodes in this cluster are recommended to have at least 100 GB of ephemeral storage.
                  uri: https://kurl.sh/docs/install-with-kurl/system-requirements
              - pass:
                  message: The nodes in this cluster have enough ephemeral storage.

Check node status​

The following example uses the nodeResources analyzer to check the status of the nodes in the cluster. The nodeResources analyzer uses data from the default clusterResources collector.

For more information, see Node Resources and Cluster Resources in the Troubleshoot documentation.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData: 
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
        collectors: []
        analyzers:
          - nodeResources:
              checkName: Node status check
              outcomes:
                - fail:
                    when: "nodeCondition(Ready) == False"
                    message: "Not all nodes are online."
                - warn:
                    when: "nodeCondition(Ready) == Unknown"
                    message: "Not all nodes are online."
                - pass:
                    message: "All nodes are online."

Collect logs using multiple selectors​

The example below uses the logs collector to collect logs from various Pods where application workloads are running. It also uses the textAnalyze collector to analyze the logs for a known error.

For more information, see Pod Logs and Regular Expression in the Troubleshoot documentation.

You can use the selector attribute of the logs collector to find Pods that have the specified labels. Depending on the complexity of an application's labeling schema, you might need a few different declarations of the logs collector, as shown in this example. You can include the logs collector as many times as needed.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData: 
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
      collectors:
        - logs:
            namespace: {{ .Release.Namespace }}
            selector:
              - app=slackernews-nginx
        - logs:
            namespace: {{ .Release.Namespace }}
            selector:
              - app=slackernews-api
        - logs:
            namespace: {{ .Release.Namespace }}
            selector:
              - app=slackernews-frontend
        - logs:
            selector:
              - app=postgres
      analyzers:      
        - textAnalyze:
            checkName: Axios Errors
            fileName: slackernews-frontend-*/slackernews.log
            regex: "error - AxiosError"
            outcomes:
              - pass:
                  when: "false"
                  message: "Axios errors not found in logs"
              - fail:
                  when: "true"
                  message: "Axios errors found in logs"

Collect logs using limits​

The example below uses the logs collector to collect Pod logs from the Pod where the application is running. This specification uses the limits field to set a maxAge and maxLines to limit the output provided.

For more information, see Pod Logs in the Troubleshoot documentation.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData: 
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
      collectors:
        - logs:
            selector:
              - app.kubernetes.io/name=myapp
            namespace: {{ .Release.Namespace }}
            limits:
              maxAge: 720h
              maxLines: 10000

Collect Prometheus metrics​

If your application runs alongside a Prometheus server, you can include the results of PromQL queries in the support bundle.

The following example uses a runPod collector to run curl against the Prometheus query API from inside the cluster. The JSON response from Prometheus is saved to prometheus-up/prometheus-up.log in the bundle, and the textAnalyze analyzer checks that the query succeeded. Update the URL to match the Prometheus service name, namespace, and port in your environment, and add a collector for each query that you want to include. Set namespace to a namespace where the support bundle can create Pods, such as your application's namespace. The Pod does not need to run in the same namespace as Prometheus because the URL uses the full service name.

This approach works for every installation method because the request is sent from a Pod in the cluster. It requires pods/create RBAC. In air-gapped environments, the curl image must be available to the cluster.

For more information, see Run Pods and Regular Expression in the Troubleshoot documentation.

apiVersion: v1
kind: Secret
metadata:
  name: prometheus-support-bundle
  labels:
    troubleshoot.sh/kind: support-bundle
stringData:
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: prometheus
    spec:
      collectors:
        - runPod:
            collectorName: prometheus-up
            name: prometheus-up
            namespace: my-app
            timeout: 60s
            podSpec:
              containers:
                - name: curl
                  image: curlimages/curl:8.16.0
                  command: ["curl"]
                  args:
                    - "-sS"
                    - "--max-time"
                    - "10"
                    - "http://prometheus-server.monitoring.svc/api/v1/query?query=up"
      analyzers:
        - textAnalyze:
            checkName: Prometheus query API
            fileName: prometheus-up/prometheus-up.log
            regex: '"status":"success"'
            outcomes:
              - pass:
                  when: "true"
                  message: The Prometheus query API responded successfully.
              - fail:
                  when: "false"
                  message: The Prometheus query did not succeed. Check `prometheus-up/prometheus-up.log` in the bundle for the curl or Prometheus error.

Alternatives to the runPod collector​

You can also query Prometheus with the http collector, but only when the Prometheus URL is reachable from wherever the support bundle is generated. The http collector sends the request from the machine or process that generates the bundle, not from a Pod in the cluster. For example, when a customer runs kubectl support-bundle from a workstation, a cluster-internal address such as http://prometheus-server.monitoring.svc does not resolve. The bundle is still generated, but the collector output contains a no such host error instead of metrics.

If the cluster runs prometheus-adapter or a similar adapter that serves the custom.metrics.k8s.io API, you can use the customMetrics collector to read metrics through the Kubernetes API without running a Pod. The collector writes each metric to metrics/<resource kind>/<metric name>/ in the bundle. The metric names that are available depend on the adapter configuration. To list them, run kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1. For more information, see Kubernetes Custom Metrics in the Troubleshoot documentation.

spec:
  collectors:
    - customMetrics:
        metricRequests:
          - namespace: my-app
            resourceMetricName: pods/kube_pod_container_status_restarts
note

The customMetrics collector is not related to the Replicated SDK custom metrics feature. For information about sending custom metrics to the Vendor Portal, see Configure custom metrics.

Collect Redis and MySQL server information​

The following example uses the redis and mysql collectors to collect information about Redis and MySQL servers running in the cluster.

For more information, see Redis and MySQL in the Troubleshoot documentation.

important

The mysql, postgres, mssql, and redis collectors accept credentials only as a literal string in the uri field. Helm template functions and KOTS config options do not protect the credential. These functions and config options resolve to the live credential when the spec is rendered and deployed to the cluster, where a support bundle can collect the credential in plain text. The built-in redactors do not reliably mask these values.

To pass a credential to a collector without exposing it, use a runPod collector with env.valueFrom.secretKeyRef. For more information, see Pass a credential to the pod in Example support bundle specs.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData: 
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
      collectors:
        - mysql:
            collectorName: mysql
            uri: 'root:<PASSWORD>@tcp(localhost:3306)/mysql'
            parameters:
              - character_set_server
              - collation_server
              - init_connect
              - innodb_file_format
              - innodb_large_prefix
              - innodb_strict_mode
              - log_bin_trust_function_creators
        - redis:
            collectorName: my-redis
            uri: rediss://default:<PASSWORD>@server:6380

Run and analyze a pod​

The example below uses the textAnalyze analyzer to check that a command successfully executes in a Pod running in the cluster. The Pod specification is defined in the runPod collector.

For more information, see Run Pods and Regular Expression in the Troubleshoot documentation.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData: 
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
      collectors:
        - runPod:
            collectorName: "static-hi"
            podSpec:
              containers:
              - name: static-hi
                image: alpine:3
                command: ["echo", "hi static!"]
      analyzers:
        - textAnalyze:
            checkName: Said hi!
            fileName: /static-hi.log
            regex: 'hi static'
            outcomes:
              - fail:
                  message: Didn't say hi.
              - pass:
                  message: Said hi!

Pass a credential to the pod​

If a collector needs a credential, such as a database password, use a runPod collector and pass the credential to the Pod with env.valueFrom.secretKeyRef. The kubelet resolves the credential when the Pod starts, so the credential never appears in the spec.

The following example tests a PostgreSQL connection. The PGPASSWORD environment variable is read from the my-app-postgres Secret, and the textAnalyze analyzer checks the output that the Pod writes to /postgres-connection.log in the bundle. The namespace field defaults to default, so set it to the namespace where the Secret exists.

important

The runPod collector saves everything that the Pod writes to stdout into the support bundle. Make sure that the command does not print the credential. For example, do not echo the environment variable or run a client with verbose or debug logging enabled.

apiVersion: v1
kind: Secret
metadata:
  name: example
  labels:
    troubleshoot.sh/kind: support-bundle
stringData:
  support-bundle-spec: |-
    apiVersion: troubleshoot.sh/v1beta2
    kind: SupportBundle
    metadata:
      name: example
    spec:
      collectors:
        - runPod:
            collectorName: postgres-connection
            # the Pod must run in the namespace where the Secret exists
            namespace: '{{ .Release.Namespace }}'
            timeout: 60s
            podSpec:
              restartPolicy: Never
              containers:
              - name: postgres-connection
                image: postgres:16-alpine
                command: ["/bin/sh", "-c"]
                # the command must not print the credential
                args:
                  - psql --quiet --no-password --command 'SELECT 1;' && echo "connection succeeded"
                env:
                - name: PGHOST
                  value: my-app-postgres
                - name: PGUSER
                  value: my-app
                - name: PGDATABASE
                  value: my-app
                # the spec contains only the Secret name and key, not the password
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: my-app-postgres
                      key: password
      analyzers:
        - textAnalyze:
            checkName: PostgreSQL connection
            fileName: /postgres-connection.log
            regex: 'connection succeeded'
            outcomes:
              - fail:
                  message: Could not connect to the PostgreSQL database.
              - pass:
                  message: Connected to the PostgreSQL database.

You can use the same pattern in preflight checks, but the Secret must already exist in the cluster when the preflight checks run. For more information, see Use a credential in a preflight check in Define preflight checks.

On this page