Skip to main content

Troubleshoot Embedded Cluster (Beta)

This topic provides information about troubleshooting Replicated Embedded Cluster installations.

For more information about Embedded Cluster, including built-in extensions and architecture, see Embedded Cluster Overview.

Troubleshoot with support bundles​

This section includes information about how to collect support bundles for Embedded Cluster installations. For more information about support bundles, see About Preflight Checks and Support Bundles.

About the default Embedded Cluster support bundle spec​

Embedded Cluster includes a default support bundle spec that collects both host- and cluster-level information:

  • The host-level information is useful for troubleshooting failures related to host configuration like DNS, networking, or storage problems.
  • Cluster-level information includes details about the components provided by Replicated, such as the Admin Console and Embedded Cluster Operator that manage install and upgrade operations. If the cluster has not installed successfully and cluster-level information is not available, then it is excluded from the bundle.

Support bundles also include application details from custom support bundle specs:

  • Embedded Cluster v3: Deploy each custom spec in a Kubernetes Secret labeled troubleshoot.sh/kind: support-bundle through your Helm chart. Embedded Cluster discovers these specs in the cluster when generating a bundle. Standalone kind: SupportBundle files included in the application release are not collected. For a Secret example, see Helm and troubleshoot template example.
  • Embedded Cluster v2: Custom support bundle specs included in the application release are collected.

For more information, see Combine support bundle specs.

Generate a support bundle​

The support-bundle command uses the default Embedded Cluster support bundle spec to collect both cluster- and host-level information. It also automatically includes any application-specific support bundle specs in the generated bundle.

To generate a support bundle:

  1. SSH onto a controller node.

    note

    You can SSH onto a worker node to generate a support bundle that contains information specific to that node. However, when run on a worker node, the support-bundle command does not capture cluster-wide information.

  2. Run the following command:

    sudo ./APP_SLUG support-bundle

    Where APP_SLUG is the unique slug for the application.

Inspect a support bundle​

After you have a bundle, upload it to the Vendor Portal for analysis. You can also inspect the bundle locally with sbctl, which lets you run kubectl commands against the state of the cluster when the bundle was collected. For more information, see Inspect Support Bundles.

View logs​

You can view logs for both Embedded Cluster and the k0s systemd service to help troubleshoot Embedded Cluster deployments.

View installation logs for Embedded Cluster​

The Embedded Cluster daemon serves the installer UI and the admin console, and writes its logs to the systemd journal. To view installation logs for Embedded Cluster:

  1. SSH onto the node where you ran install or upgrade.

  2. Run the following command, where APP_SLUG is the unique slug for the application:

    sudo journalctl -u APP_SLUG

In Embedded Cluster versions earlier than 3.17.0-beta.1, installation logs are written to .log files in /var/log/APP_SLUG.

View K0s logs​

You can use the journalctl command line tool to access logs for systemd services, including k0s. For more information about k0s, see the k0s documentation.

To use journalctl to view k0s logs:

  1. SSH onto a controller node or a worker node.

  2. Use journalctl to view logs for the k0s systemd service that Embedded Cluster deployed.

    Examples:

    journalctl -u k0scontroller
    journalctl -u k0sworker

Access the cluster​

When troubleshooting, it can be useful to list the cluster and view logs using the kubectl command line tool. For additional suggestions related to troubleshooting applications, see Troubleshooting Applications in the Kubernetes documentation.

To access the cluster and use other included binaries:

  1. SSH into a controller node.

    note

    You cannot run the shell command on worker nodes.

  2. Use the Embedded Cluster shell command to start a shell with access to the cluster:

    sudo ./APP_SLUG shell

    Where APP_SLUG is the unique slug for the application.

    The output looks similar to the following:

       __4___
    _  \ \ \ \   Welcome to APP_SLUG debug shell.
    <'\ /_/_/_/   This terminal is now configured to access your cluster.
    ((____!___/) Type 'exit' (or CTRL+d) to exit.
     \0\0\0\0\/  Happy hacking.
    ~~~~~~~~~~~
    root@alex-ec-1:/home/alex# export KUBECONFIG="/var/lib/embedded-cluster/k0s/pki/admin.conf"
    root@alex-ec-1:/home/alex# export PATH="$PATH:/var/lib/embedded-cluster/bin"
    root@alex-ec-1:/home/alex# source <(k0s completion bash)
    root@alex-ec-1:/home/alex# source <(cat /var/lib/embedded-cluster/bin/kubectl_completion_bash.sh)
    root@alex-ec-1:/home/alex# source /etc/bash_completion

    The appropriate kubeconfig is exported, and the location of useful binaries like kubectl and Replicated’s preflight and support-bundle plugins is added to PATH.

  3. Use the available binaries as needed.

    Example:

    kubectl version
    Client Version: v1.29.1
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.29.1+k0s

  4. Type exit or Ctrl + D to exit the shell.

Troubleshoot errors​

This section provides troubleshooting advice for common errors.

Installation failure when a release includes the NVIDIA gpu operator as a Helm extension​

Symptom​

A release that includes that includes the NVIDIA GPU Operator as a Helm extension fails to install.

Cause​

If there are any containerd services on the host, the NVIDIA GPU Operator will generate an invalid containerd config, causing the installation to fail.

This is the result of a known issue with v24.9.x of the NVIDIA GPU Operator. For more information about the known issue, see container-toolkit does not modify the containerd config correctly when there are multiple instances of the containerd binary in the nvidia-container-toolkit repository in GitHub.

For more information about including the GPU Operator as a Helm extension, see NVIDIA GPU Operator in Configure Embedded Cluster.

Solution​

To troubleshoot:

  1. Remove any existing containerd services that are running on the host (such as those deployed by Docker).

  2. Reset and reboot the node:

    sudo ./APP_SLUG reset

    Where APP_SLUG is the unique slug for the application.

    For more information, see Reset a Machine in Access and Manage Embedded Clusters.

  3. Re-install with Embedded Cluster.

Calico networking issues​

Symptom​

Symptoms of Calico networking issues can include:

  • The pod enters a CrashLoopBackOff state with failed health checks:

    Warning Unhealthy 6h51m (x3 over 6h52m) kubelet Liveness probe failed: Get "http://<ip:port>/readyz": dial tcp <ip:port>: connect: no route to host
    Warning Unhealthy 6h51m (x19 over 6h52m) kubelet Readiness probe failed: Get "http://<ip:port>/readyz": dial tcp <ip:port>: connect: no route to host
    ....
    Unhealthy               pod/registry-dc699cbcf-pkkbr     Readiness probe failed: Get "https://<ip:port>/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    Unhealthy               pod/registry-dc699cbcf-pkkbr     Liveness probe failed: Get "https://<ip:port>/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
    ...

  • The pod log contains an I/O timeout:

    server APIs: config.k8ssandra.io/v1beta1: Get \"https://***HIDDEN***:443/apis/config.k8ssandra.io/v1beta1\": dial tcp ***HIDDEN***:443: i/o timeout"}

Cause​

Reasons can include:

  • Pod CIDR and service CIDR overlap with the host network CIDR.

  • Incorrect kernel parameters values.

  • VXLAN traffic getting dropped. By default, Calico uses VXLAN as the overlay networking protocol, with Always mode. This mode encapsulates all pod-to-pod traffic in VXLAN packets. If for some reasons, the VXLAN packets get filtered by the network, the pod will not able to communicate with other pods.

  • A security tool on the host blocks or removes the firewall rules that cluster networking uses. See Security tools interfere with the cluster.

Solution​

To troubleshoot pod CIDR and service CIDR overlapping with the host network CIDR:

  1. Run the following command to verify the pod and service CIDR:

    cat /etc/k0s/k0s.yaml | grep -i cidr
        podCIDR: 10.244.0.0/17
        serviceCIDR: 10.244.128.0/17

    The default pod CIDR is 10.244.0.0/17 and the default service CIDR is 10.244.128.0/17.

  2. View pod network interfaces excluding Calico interfaces, and ensure there are no overlapping CIDRs.

    ip route | grep -v cali
    default via 10.152.0.1 dev ens4 proto dhcp src 10.152.0.4 metric 100
    10.152.0.1 dev ens4 proto dhcp scope link src 10.152.0.4 metric 100
    blackhole 10.244.101.192/26 proto 80
    169.254.169.254 via 10.152.0.1 dev ens4 proto dhcp src 10.152.0.4 metric 100

  3. Reset and reboot the installation:

    sudo ./APP_SLUG reset

    Where APP_SLUG is the unique slug for the application.

    For more information, see Reset a Machine in Access and Manage Embedded Clusters.

  4. Reinstall the application with different CIDRs using the --cidr flag:

    sudo ./APP_SLUG install --license license.yaml --cidr 172.16.136.0/16

    For more information, see install.

Security tools interfere with the cluster​

Symptom​

Symptoms can include:

  • The kube-proxy or calico-node pods restart repeatedly.

  • Pod logs contain I/O timeouts when connecting to the Kubernetes API service IP address. This is the first address in the service CIDR, which is 10.244.128.1 by default.

  • Embedded Cluster binaries or systemd units are missing or fail to run.

  • Containers fail to restart.

These symptoms often appear during a new installation, or after a security tool restarts, receives a policy update, or the host reboots.

Cause​

A security tool on the host is blocking or removing firewall rules, binaries, or files that Embedded Cluster needs. This includes firewall management, endpoint protection, and application control tools.

Solution​

Disable the tool on Embedded Cluster hosts. If disabling the tool is not an option, work with the security team to update the tool's policy so that it allows what Embedded Cluster needs. For more information about the requirements, see Security tools on the host in Embedded Cluster installation requirements.

Disabling a tool or updating its policy does not undo changes that the tool already made. You might also need to reboot the host to reset the firewall and restore any kernel parameters that the tool changed. In some cases, you also need to remove ebtables rules that the tool added.

Support bundles report this problem when they detect CrowdStrike Falcon or Illumio VEN on a host where the Calico firewall rules are missing. For more information, see Troubleshoot with support bundles.

Pods cannot resolve names that resolve on the host​

Symptom​

Symptoms can include:

  • Pods fail to connect to an external service, such as a database, by host name, but the same host name resolves on the host.

  • Name lookups from pods are slow or time out intermittently.

  • Only some pods fail to resolve a name, even though all nodes have the same DNS settings.

  • A change to the CoreDNS ConfigMap fixes name resolution, but the problem returns after an upgrade.

  • The CoreDNS pods restart repeatedly, and their logs contain no nameservers found.

Cause​

Common causes include the following:

  • CoreDNS has no upstream nameserver that it can reach. This happens when the file that CoreDNS forwards to lists no nameservers, or lists only a loopback address.

  • A service on the host, such as systemd-resolved or NetworkManager, manages /etc/resolv.conf and overwrites manual edits to it.

  • The host blocks pod traffic. If IP forwarding is disabled (net.ipv4.ip_forward is 0), or host firewall rules drop traffic from pods, DNS queries from pods fail even when the DNS configuration is correct.

  • The host has many DNS search domains. Pods inherit them, so each lookup from a pod can send a query for every search domain. Container images based on musl libc, such as Alpine images, handle search domains differently from other images, and can fail to resolve a name that resolves in other pods.

  • A pod's DNS settings are written when the pod is created. Restarting a container keeps the old settings, so a pod created before a change to the host's DNS settings keeps the old settings until the pod is recreated.

  • Embedded Cluster manages the CoreDNS configuration and does not preserve direct edits to the CoreDNS ConfigMap. An Embedded Cluster upgrade that changes the default CoreDNS configuration replaces the ConfigMap, and any edits to it are lost.

For more information about which file CoreDNS forwards to and how pods inherit the host's DNS settings, see DNS on the host in Embedded Cluster installation requirements.

Solution​

To resolve DNS issues in pods, try the following:

  • If CoreDNS logs no nameservers found, configure the host's DNS servers through the service that manages them, so that the file CoreDNS forwards to lists at least one nameserver that pods can reach. Do not edit a file that a service manages, because the service overwrites the edit. On hosts that use systemd-resolved, resolvectl status lists the DNS servers that it knows about. Then recreate the CoreDNS pods:

    kubectl -n kube-system rollout restart deployment coredns

    Embedded Cluster chooses which file CoreDNS forwards to when it starts on the node. If your change affects that choice, such as enabling or disabling systemd-resolved, or adding or removing nameservers alongside 127.0.0.53, reboot the node before you recreate the CoreDNS pods.

  • Run sysctl net.ipv4.ip_forward on the host and confirm that it returns 1. If a security tool or host firewall manages the host, see Security tools interfere with the cluster in this document.

  • To check the search domains that an affected pod uses, run cat /etc/resolv.conf in the pod. Pods with the ClusterFirst DNS policy, which is the Kubernetes default, list the cluster's search domains followed by the host's. If the host's search domains are out of date, recreate the pod by deleting it or by restarting its workload, for example with kubectl rollout restart. Restarting a container does not update the settings.

  • If the host has many search domains, remove the ones that the host does not need. Alternatively, the application vendor can lower ndots for the affected workloads using dnsConfig in the pod spec. For more information about setting dnsConfig, see Pod's DNS Config in the Kubernetes documentation.

  • If a name must resolve through a private DNS zone, configure the host's nameservers to resolve that zone. CoreDNS forwards these names to the host's nameservers, so the change persists across reboots and upgrades. Do not rely on edits to the CoreDNS ConfigMap.

On this page