Using Embedded Cluster

This topic describes how to use the Replicated Embedded Cluster to configure, install, and manage your application in an embedded Kubernetes cluster.

note

If you are instead looking for information about creating Kubernetes Installers with Replicated kURL, see the Replicated kURL section.

Overview

Replicated Embedded Cluster allows you to distribute a Kubernetes cluster and your application together as a single appliance, making it easy for enterprise users to install, update, and manage the application and the cluster in tandem. Embedded Cluster is based on the open source Kubernetes distribution k0s. For more information, see the k0s documentation.

For software vendors, Embedded Cluster provides a Config for defining characteristics of the cluster that will be created in the customer environment. Additionally, each version of Embedded Cluster includes a specific version of Replicated KOTS, ensuring compatibility between KOTS and the cluster. For enterprise users, cluster updates are done automatically at the same time as application updates, allowing users to more easily keep the cluster up-to-date without needing to use kubectl.

The following diagram demonstrates how Kubernetes and an application are installed into a customer environment using Embedded Cluster:

View a larger version of this image

As shown in the diagram above, the Embedded Cluster Config is included in the application release in the Replicated Vendor Portal and is used to generate the Embedded Cluster installation assets. Users can download these installation assets from the Replicated app service (replicated.app) on the command line, then run the Embedded Cluster installation command to install Kubernetes and the KOTS Admin Console. Finally, users access the Admin Console to optionally add nodes to the cluster and to configure and install the application.

Comparison to kURL

Embedded Cluster is a successor to Replicated kURL. Compared to kURL, Embedded Cluster offers several improvements such as:

• Significantly faster installation, updates, and node joins
• A redesigned Admin Console UI for managing the cluster
• Improved support for multi-node clusters
• One-click updates of both the application and the cluster at the same time

Requirements

System Requirements

• Linux operating system

• x86-64 architecture

• systemd

• At least 2GB of memory and 2 CPU cores

• The disk on the host must have a maximum P99 write latency of 10 ms. This supports etcd performance and stability. For more information about the disk write latency requirements for etcd, see Disks in Hardware recommendations and What does the etcd warning “failed to send out heartbeat on time” mean? in the etcd documentation.

• The filesystem at /var/lib/embedded-cluster has 40Gi or more of total space and must be less than 80% full

  note

  The directory used for data storage can be changed by passing the --data-dir flag with the Embedded Cluster install command. For more information, see Embedded Cluster Install Command Options.

  Note that in addition to the primary /var/lib/embedded-cluster directory, Embedded Cluster creates directories and files in the following locations:

  • /etc/cni
  • /etc/k0s
  • /opt/cni
  • /opt/containerd
  • /run/calico
  • /run/containerd
  • /run/k0s
  • /sys/fs/cgroup/kubepods
  • /sys/fs/cgroup/system.slice/containerd.service
  • /sys/fs/cgroup/system.slice/k0scontroller.service
  • /usr/libexec/k0s
  • /var/lib/calico
  • /var/lib/cni
  • /var/lib/containers
  • /var/lib/kubelet
  • /var/log/calico
  • /var/log/containers
  • /var/log/pods
  • /usr/local/bin/k0s

• (Online installations only) Access to replicated.app and proxy.replicated.com or your custom domain for each

• Embedded Cluster is based on k0s, so all k0s system requirements and external runtime dependencies apply. See System requirements and External runtime dependencies in the k0s documentation.

Port Requirements

Embedded Cluster requires that the following ports are open and available:

• 2379/TCP *
• 2380/TCP
• 4789/UDP
• 6443/TCP
• 7443/TCP
• 9091/TCP
• 9099/TCP *
• 9443/TCP
• 10248/TCP *
• 10249/TCP
• 10250/TCP
• 10256/TCP
• 10257/TCP *
• 10259/TCP *
• 30000/TCP ***
• 50000/TCP * ** ***

* These ports are used only by processes running on the same node. Ensure that there are no other processes using them. It is not necessary to create firewall openings for these ports.

** Required for air gap installations only.

*** By default, the Admin Console and Local Artifact Mirror (LAM) run on ports 30000 and 50000, respectively. If these ports are occupied, you can select different ports during installation. For more information, see Embedded Cluster Install Command Options.

Limitations

Embedded Cluster has the following limitations:

• Reach out about migrating from kURL: We are helping several customers migrate from kURL to Embedded Cluster. Reach out to Alex Parker at alexp@replicated.com for more information.

• Multi-node support is in beta: Support for multi-node embedded clusters is in beta, and enabling high availability for multi-node clusters is in alpha. Only single-node embedded clusters are generally available. For more information, see Managing Multi-Node Clusters with Embedded Cluster.

• Disaster recovery is in alpha: Disaster Recovery for Embedded Cluster installations is in alpha. For more information, see Disaster Recovery for Embedded Cluster (Alpha).

• Partial rollback support: In Embedded Cluster 1.17.0 and later, rollbacks are supported only when rolling back to a version where there is no change to the Embedded Cluster Config compared to the currently-installed version. For example, users can roll back to release version 1.0.0 after upgrading to 1.1.0 only if both 1.0.0 and 1.1.0 use the same Embedded Cluster Config. For more information about how to enable rollbacks for your application in the KOTS Application custom resource, see allowRollback in Application.

• Changing node hostnames is not supported: After a host is added to a Kubernetes cluster, Kubernetes assumes that the hostname and IP address of the host will not change. If you need to change the hostname or IP address of a node, you must first remove the node from the cluster. For more information about the requirements for naming nodes, see Node name uniqueness in the Kubernetes documentation.

• Automatic updates not supported: Configuring automatic updates from the Admin Console so that new versions are automatically deployed is not supported for Embedded Cluster installations. For more information, see Configuring Automatic Updates.

• Embedded Cluster installation assets not available through the Download Portal: The assets required to install with Embedded Cluster cannot be shared with users through the Download Portal. Users can follow the Embedded Cluster installation instructions to download and extract the installation assets. For more information, see Online Installation with Embedded Cluster.

• minKotsVersion and targetKotsVersion not supported: The minKotsVersion and targetKotsVersion fields in the KOTS Application custom resource are not supported for Embedded Cluster installations. This is because each version of Embedded Cluster includes a particular version of KOTS. Setting targetKotsVersion or minKotsVersion to a version of KOTS that does not coincide with the version that is included in the specified version of Embedded Cluster will cause Embedded Cluster installations to fail with an error message like: Error: This version of App Name requires a different version of KOTS from what you currently have installed. To avoid installation failures, do not use targetKotsVersion or minKotsVersion in releases that support installation with Embedded Cluster.

• Support bundles over 100MB in the Admin Console: Support bundles are stored in rqlite. Bundles over 100MB could cause rqlite to crash, causing errors in the installation. You can still generate a support bundle from the command line. For more information, see Generating Support Bundles for Embedded Cluster.

• Kubernetes version template functions not supported: The KOTS KubernetesVersion, KubernetesMajorVersion, and KubernetesMinorVersion template functions do not provide accurate Kubernetes version information for Embedded Cluster installations. This is because these template functions are rendered before the Kubernetes cluster has been updated to the intended version. However, KubernetesVersion is not necessary for Embedded Cluster because vendors specify the Embedded Cluster version, which includes a known Kubernetes version.

• Custom domains not supported: Embedded Cluster does not support the use of custom domains, even if custom domains are configured. We intend to add support for custom domains. For more information about custom domains, see About Custom Domains.

• KOTS Auto-GitOps workflow not supported: Embedded Cluster does not support the KOTS Auto-GitOps workflow. If an end-user is interested in GitOps, consider the Helm install method instead. For more information, see Installing with Helm.

• Downgrading Kubernetes not supported: Embedded Cluster does not support downgrading Kubernetes. The admin console will not prevent end-users from attempting to downgrade Kubernetes if a more recent version of your application specifies a previous Embedded Cluster version. You must ensure that you do not promote new versions with previous Embedded Cluster versions.

• Templating not supported in Embedded Cluster Config: The Embedded Cluster Config resource does not support the use of Go template functions, including KOTS template functions.

• Policy enforcement on Embedded Cluster workloads is not supported: The Embedded Cluster runs workloads that require higher levels of privilege. If your application installs a policy enforcement engine such as Gatekeeper or Kyverno, ensure that its policies are not enforced in the namespaces used by Embedded Cluster.

• Installing on STIG- and CIS-hardened OS images is not supported: Embedded Cluster isn't tested on these images, and issues have arisen when trying to install on them.

Quick Start

You can use the following steps to get started quickly with Embedded Cluster. More detailed documentation is available below.

1. Create a new customer or edit an existing customer and select the Embedded Cluster Enabled license option. Save the customer.

2. Create a new release that includes your application. In that release, create an Embedded Cluster Config that includes, at minimum, the Embedded Cluster version you want to use. See the Embedded Cluster GitHub repo to find the latest version.

   Example Embedded Cluster Config:

   apiVersion: embeddedcluster.replicated.com/v1beta1
   kind: Config
   spec:
     version: 1.19.0+k8s-1.30

3. Save the release and promote it to the channel the customer is assigned to.

4. Return to the customer page where you enabled Embedded Cluster. At the top right, click Install instructions and choose Embedded Cluster. A dialog appears with instructions on how to download the Embedded Cluster installation assets and install your application.

   

   View a larger version of this image

5. On your VM, run the commands in the Embedded Cluster install instructions dialog.

   

   View a larger version of this image

6. Enter an Admin Console password when prompted.

   The Admin Console URL is printed when the installation finishes. Access the Admin Console to begin installing your application. During the installation process in the Admin Console, you have the opportunity to add nodes if you want a multi-node cluster. Then you can provide application config, run preflights, and deploy your application.

About Configuring Embedded Cluster

To install an application with Embedded Cluster, an Embedded Cluster Config must be present in the application release. The Embedded Cluster Config lets you define several characteristics about the cluster that will be created.

For more information, see Embedded Cluster Config.

About Installing with Embedded Cluster

This section provides an overview of installing applications with Embedded Cluster.

Installation Options

Embedded Cluster supports installations in online (internet-connected) environments and air gap environments with no outbound internet access.

For online installations, Embedded Cluster also supports installing behind a proxy server.

For more information about how to install with Embedded Cluster, see:

• Online Installation wtih Embedded Cluster
• Air Gap Installation with Embedded Cluster

Customer-Specific Installation Instructions

To install with Embedded Cluster, you can follow the customer-specific instructions provided on the Customer page in the Vendor Portal. For example:

View a larger version of this image

(Optional) Serve Installation Assets Using the Vendor API

To install with Embedded Cluster, you need to download the Embedded Cluster installer binary and a license. Air gap installations also require an air gap bundle. Some vendors already have a portal where their customers can log in to access documentation or download artifacts. In cases like this, you can serve the Embedded Cluster installation essets yourself using the Replicated Vendor API, rather than having customers download the assets from the Replicated app service using a curl command during installation.

To serve Embedded Cluster installation assets with the Vendor API:

1. If you have not done so already, create an API token for the Vendor API. See Using the Vendor API v3.

2. Call the Get an Embedded Cluster release endpoint to download the assets needed to install your application with Embedded Cluster. Your customers must take this binary and their license and copy them to the machine where they will install your application.

   Note the following:

   • (Recommended) Provide the customerId query parameter so that the customer’s license is included in the downloaded tarball. This mirrors what is returned when a customer downloads the binary directly using the Replicated app service and is the most useful option. Excluding the customerId is useful if you plan to distribute the license separately.

   • If you do not provide any query parameters, this endpoint downloads the Embedded Cluster binary for the latest release on the specified channel. You can provide the channelSequence query parameter to download the binary for a particular release.

About Host Preflight Checks

During installation, Embedded Cluster automatically runs a default set of host preflight checks. The default host preflight checks are designed to verify that the installation environment meets the requirements for Embedded Cluster, such as:

• The system has sufficient disk space
• The system has at least 2G of memory and 2 CPU cores
• The system clock is synchronized

For Embedded Cluster requirements, see Requirements. For the full default host preflight spec for Embedded Cluster, see host-preflight.yaml in the embedded-cluster repository in GitHub.

If any of the host preflight checks fail, installation is blocked and a message describing the failure is displayed. For more information about host preflight checks for installations on VMs or bare metal servers, see About Host Preflights.

Limitations

Embedded Cluster host preflight checks have the following limitations:

• The default host preflight checks for Embedded Cluster cannot be modified, and vendors cannot provide their own custom host preflight spec for Embedded Cluster.
• Host preflight checks do not check that any application-specific requirements are met. For more information about defining preflight checks for your application, see Defining Preflight Checks.

Skip Host Preflight Checks

You can skip host preflight checks by passing the --skip-host-preflights flag with the Embedded Cluster install command. For example:

sudo ./my-app install --license license.yaml --skip-host-preflights

When you skip host preflight checks, the Admin Console still runs any application-specific preflight checks that are defined in the release before the application is deployed.

note

Skipping host preflight checks is not recommended for production installations.

About Managing Multi-Node Clusters with Embedded Cluster

This section describes managing nodes in multi-node clusters created with Embedded Cluster.

Defining Node Roles for Multi-Node Clusters

You can optionally define node roles in the Embedded Cluster Config. For multi-node clusters, roles can be useful for the purpose of assigning specific application workloads to nodes. If nodes roles are defined, users access the Admin Console to assign one or more roles to a node when it is joined to the cluster.

For more information, see roles in Embedded Cluster Config.

Adding Nodes

Users can add nodes to a cluster with Embedded Cluster from the Admin Console. The Admin Console provides the join command used to add nodes to the cluster.

For more information, see Managing Multi-Node Clusters with Embedded Cluster.

High Availability for Multi-Node Clusters (Alpha)

Multi-node clusters are not highly available by default. Enabling high availability (HA) requires that at least three controller nodes are present in the cluster. Users can enable HA when joining the third node.

For more information about creating HA multi-node clusters with Embedded Cluster, see Enable High Availability for Multi-Node Clusters (Alpha) in Managing Multi-Node Clusters with Embedded Cluster.

About Performing Updates with Embedded Cluster

When you update an application installed with Embedded Cluster, you update both the application and the cluster infrastructure together, including Kubernetes, KOTS, and other components running in the cluster. There is no need or mechanism to update the infrastructure on its own.

When you deploy a new version, any changes to the cluster are deployed first. The Admin Console waits until the cluster is ready before updatng the application.

Any changes made to the Embedded Cluster Config, including changes to the Embedded Cluster version, Helm extensions, and unsupported overrides, trigger a cluster update.

When performing an upgrade with Embedded Cluster, the user is able to change the application config before deploying the new version. Additionally, the user's license is synced automatically. Users can also make config changes and sync their license outside of performing an update. This requires deploying a new version to apply the config change or license sync.

For more information about updating, see Performing Updates with Embedded Cluster.

Access the Cluster

With Embedded Cluster, end-users are rarely supposed to need to use the CLI. Typical workflows, like updating the application and the cluster, are driven through the Admin Console.

Nonetheless, there are times when vendors or their customers need to use the CLI for development or troubleshooting.

To access the cluster and use other included binaries:

1. SSH onto a controller node.

2. Use the Embedded Cluster shell command to start a shell with access to the cluster:

   sudo ./APP_SLUG shell

   The output looks similar to the following:

      __4___
   _  \ \ \ \   Welcome to APP_SLUG debug shell.
   <'\ /_/_/_/   This terminal is now configured to access your cluster.
   ((____!___/) Type 'exit' (or CTRL+d) to exit.
   \0\0\0\0\/  Happy hacking.
   ~~~~~~~~~~~
   root@alex-ec-2:/home/alex# export KUBECONFIG="/var/lib/embedded-cluster/k0s/pki/admin.conf"
   root@alex-ec-2:/home/alex# export PATH="$PATH:/var/lib/embedded-cluster/bin"
   root@alex-ec-2:/home/alex# source <(kubectl completion bash)
   root@alex-ec-2:/home/alex# source /etc/bash_completion

   The appropriate kubeconfig is exported, and the location of useful binaries like kubectl and Replicated’s preflight and support-bundle plugins is added to PATH.

   note

   You cannot run the shell command on worker nodes.

3. Use the available binaries as needed.

   Example:

   kubectl version

   Client Version: v1.29.1
   Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
   Server Version: v1.29.1+k0s

4. Type exit or Ctrl + D to exit the shell.

   note

   If you encounter a typical workflow where your customers have to use the Embedded Cluster shell, reach out to Alex Parker at alexp@replicated.com. These workflows might be candidates for additional Admin Console functionality.

Reset a Node

Resetting a node removes the cluster and your application from that node. This is useful for iteration, development, and when mistakes are made, so you can reset a machine and reuse it instead of having to procure another machine.

If you want to completely remove a cluster, you need to reset each node individually.

When resetting a node, OpenEBS PVCs on the node are deleted. Only PVCs created as part of a StatefulSet will be recreated automatically on another node. To recreate other PVCs, the application will need to be redeployed.

To reset a node:

1. SSH onto the machine. Ensure that the Embedded Cluster binary is still available on that machine.

2. Run the following command to reset the node and automatically reboot the machine to ensure that transient configuration is also reset:

   sudo ./APP_SLUG reset

   Where APP_SLUG is the unique slug for the application.

   note

   Pass the --no-prompt flag to disable interactive prompts. Pass the --force flag to ignore any errors encountered during the reset.

Additional Use Cases

This section outlines some additional use cases for Embedded Cluster. These are not officially supported features from Replicated, but are ways of using Embedded Cluster that we or our customers have experimented with that might be useful to you.

NVIDIA GPU Operator

The NVIDIA GPU Operator uses the operator framework within Kubernetes to automate the management of all NVIDIA software components needed to provision GPUs. For more information about this operator, see the NVIDIA GPU Operator documentation. You can include the operator in your release as an additional Helm chart, or using the Embedded Cluster Helm extensions. For information about Helm extensions, see extensions in Embedded Cluster Config.

Using this operator with Embedded Cluster requires configuring the containerd options in the operator as follows:

toolkit:
   env:
   - name: CONTAINERD_CONFIG
     value: /etc/k0s/containerd.d/nvidia.toml
   - name: CONTAINERD_SOCKET
     value: /run/k0s/containerd.sock

Troubleshoot with Support Bundles

Embedded Cluster includes a default support bundle spec that collects both host- and cluster-level information.

The host-level information is useful for troubleshooting failures related to host configuration like DNS, networking, or storage problems. Cluster-level information includes details about the components provided by Replicated, such as the Admin Console and Embedded Cluster operator that manage install and upgrade operations. If the cluster has not installed successfully and cluster-level information is not available, then it is excluded from the bundle.

In addition to the host- and cluster-level details provided by the default Embedded Cluster spec, support bundles generated for Embedded Cluster installations also include app-level details provided by any custom support bundle specs that you included in the application release.

There are different steps to generate a support bundle depending on the version of Embedded Cluster installed.

For Versions 1.17.0 and Later

For Embedded Cluster 1.17.0 and later, you can run the Embedded Cluster support-bundle command to generate a support bundle.

The support-bundle command uses the default Embedded Cluster support bundle spec to collect both cluster- and host-level information. It also automatically includes any application-specific support bundle specs in the generated bundle.

To generate a support bundle:

1. SSH onto a controller node.

   note

   You can SSH onto a worker node to generate a support bundle that contains information specific to that node. However, when run on a worker node, the support-bundle command does not capture cluster-wide information.

2. Run the following command:

   sudo ./APP_SLUG support-bundle

   Where APP_SLUG is the unique slug for the application.

For Versions Earlier Than 1.17.0

For Embedded Cluster versions earlier than 1.17.0, you can generate a support bundle from the shell using the kubectl support-bundle plugin.

To generate a bundle with the support-bundle plugin, you pass the default Embedded Cluster spec to collect both cluster- and host-level information. You also pass the --load-cluster-specs flag, which discovers all support bundle specs that are defined in Secrets or ConfigMaps in the cluster. This ensures that any application-specific specs are also included in the bundle. For more information, see Discover Cluster Specs in the Troubleshoot documentation.

To generate a bundle:

1. SSH onto a controller node.

2. Use the Embedded Cluster shell command to start a shell with access to the cluster:

   sudo ./APP_SLUG shell

   Where APP_SLUG is the unique slug for the application.

   The output looks similar to the following:

       __4___
   _  \ \ \ \   Welcome to APP_SLUG debug shell.
   <'\ /_/_/_/   This terminal is now configured to access your cluster.
   ((____!___/) Type 'exit' (or CTRL+d) to exit.
   \0\0\0\0\/  Happy hacking.
   ~~~~~~~~~~~
   root@alex-ec-2:/home/alex# export KUBECONFIG="/var/lib/embedded-cluster/k0s/pki/admin.conf"
   root@alex-ec-2:/home/alex# export PATH="$PATH:/var/lib/embedded-cluster/bin"
   root@alex-ec-2:/home/alex# source <(kubectl completion bash)
   root@alex-ec-2:/home/alex# source /etc/bash_completion

   The appropriate kubeconfig is exported, and the location of useful binaries like kubectl and the preflight and support-bundle plugins is added to PATH.

   note

   The shell command cannot be run on non-controller nodes.

3. Generate the support bundle using the default Embedded Cluster spec and the --load-cluster-specs flag:

   kubectl support-bundle  --load-cluster-specs /var/lib/embedded-cluster/support/host-support-bundle.yaml