Embedded Cluster installation requirements

This topic lists the installation requirements for Replicated Embedded Cluster. Ensure that the installation environment meets these requirements before attempting to install.

System requirements

• Linux operating system

• cgroups v2 (required for Kubernetes versions 1.35 and later)

• x86-64 architecture

• systemd

• At least 2GB of memory and 2 CPU cores

• The disk on the host must have a maximum P99 write latency of 10 ms. This supports etcd performance and stability. For more information about the disk write latency requirements for etcd, see Disks in Hardware recommendations and What does the etcd warning “failed to send out heartbeat on time” mean? in the etcd documentation.

• The user performing the installation must have root access to the machine, such as with sudo.

• The data directory used by Embedded Cluster must have 40Gi or more of total space and be less than 80% full. The data directory cannot be on a filesystem mounted with the noexec option, because Embedded Cluster runs binaries from it. By default, the data directory is /var/lib/embedded-cluster. The directory can be changed by passing the --data-dir flag with the Embedded Cluster install command. For more information, see install.

  By default, new Embedded Cluster installations store pod logs in <data-dir>/k0s/pod-logs. Pod log volume counts toward the required space in the data directory. Storing pod logs in this directory lets kubelet account for their disk usage when it manages disk pressure and pod eviction.

  Note that in addition to the primary data directory, Embedded Cluster creates directories and files in the following locations:

  • /etc/cni
  • /etc/k0s
  • /opt/cni
  • /opt/containerd
  • /run/calico
  • /run/containerd
  • /run/k0s
  • /sys/fs/cgroup/kubepods
  • /sys/fs/cgroup/system.slice/containerd.service
  • /sys/fs/cgroup/system.slice/k0scontroller.service
  • /usr/libexec/k0s
  • /var/lib/calico
  • /var/lib/cni
  • /var/lib/containers
  • /var/lib/kubelet
  • /var/log/calico
  • /var/log/containers
  • /var/log/embedded-cluster
  • /var/log/pods (installations created before Embedded Cluster 2.19.8)
  • /usr/local/bin/k0s

• (Online installations only) Access to replicated.app and proxy.replicated.com or your custom domain for each

• Embedded Cluster is based on k0s, so all k0s system requirements and external runtime dependencies apply. See System requirements and External runtime dependencies in the k0s documentation.

Host configuration

Embedded Cluster configures the following kernel settings itself at the start of installation, and when a node joins the cluster, before the host preflight checks run. The preflight checks then verify the result and report anything that could not be applied. You do not need to set these values in advance. This section describes what Embedded Cluster changes on the host, which is often needed for security review.

Every change in this section is best effort. If one cannot be applied, installation continues and the host preflight checks report the problem. Embedded Cluster uses the modprobe and sysctl commands to make these changes. If either command is missing from the host, Embedded Cluster skips the related settings.

Kernel modules. Written to /etc/modules-load.d/99-embedded-cluster.conf so that they persist across reboots, and loaded immediately if the host kernel provides them:

overlay, ip_tables, nf_tables, nft_compat, br_netfilter, and nf_conntrack

Network parameters. Written to /etc/sysctl.d/99-embedded-cluster.conf and applied:

Parameter	Value
net.ipv4.ip_forward	1
net.ipv4.conf.all.forwarding	1
net.ipv4.conf.default.forwarding	1
net.ipv6.conf.all.forwarding	1
net.ipv6.conf.default.forwarding	1
net.bridge.bridge-nf-call-iptables	1
net.bridge.bridge-nf-call-ip6tables	1
net.ipv4.conf.all.arp_filter	0
net.ipv4.conf.default.arp_filter	0
net.ipv4.conf.all.arp_ignore	0
net.ipv4.conf.default.arp_ignore	0
net.ipv4.conf.all.rp_filter	2
net.ipv4.conf.default.rp_filter	2

Reverse path filtering is set to loose mode. Strict mode, which is the value 1, drops pod traffic.

Resource limits. Written to /etc/sysctl.d/99-dynamic-embedded-cluster.conf, and only for values that are lower on the host than the following minimums:

Parameter	Minimum value
fs.inotify.max_user_instances	1024
fs.inotify.max_user_watches	65536

Port requirements

This section lists the ports used by Embedded Cluster. These ports must be open and available for both single- and multi-node installations.

Ports used by local processes

The following ports must be open and available for use by local processes running on the same node. It is not necessary to create firewall openings for these ports.

• 2379/TCP
• 7443/TCP
• 9099/TCP
• 10248/TCP
• 10257/TCP
• 10259/TCP

Ports required for bidirectional communication between nodes

The following ports are used for bidirectional communication between nodes.

For multi-node installations, create firewall openings between nodes for these ports.

For single-node installations, ensure that there are no other processes using these ports. Although there is no communication between nodes in single-node installations, these ports are still required.

• 2380/TCP
• 4789/UDP
• 6443/TCP
• 9091/TCP
• 9443/TCP
• 10249/TCP
• 10250/TCP
• 10256/TCP

Admin Console port

The KOTS Admin Console requires that port 30000/TCP is open and available. Create a firewall opening for port 30000/TCP so that the Admin Console can be accessed by the end user.

Additionally, port 30000 must be accessible by nodes joining the cluster.

If port 30000 is occupied, you can select a different port for the Admin Console during installation. For more information, see install.

LAM port

The Local Artifact Mirror (LAM) requires that port 50000/TCP is open and available.

If port 50000 is occupied, you can select a different port for the LAM during installation. For more information, see install.

DNS on the host

Embedded Cluster runs CoreDNS to resolve names inside the cluster. CoreDNS forwards names that it cannot resolve itself, such as the host name of an external database, to the nameservers configured on the host. The host must meet the following requirements:

• /etc/resolv.conf lists at least one nameserver.

• /etc/resolv.conf does not list a loopback address, such as localhost, 127.0.0.1, or 127.0.1.1, as a nameserver. Inside the cluster, a loopback address points CoreDNS back to itself, so queries loop instead of reaching a real nameserver.

  Hosts that use systemd-resolved can list its local stub resolver, 127.0.0.53, in /etc/resolv.conf, as long as it is the only nameserver listed. In that case, Embedded Cluster uses the upstream nameservers from /run/systemd/resolve/resolv.conf instead, so that file must list at least one nameserver. If /etc/resolv.conf lists 127.0.0.53 alongside other nameservers, Embedded Cluster uses /etc/resolv.conf as it is, and CoreDNS cannot reach 127.0.0.53.

The host preflight checks fail if /etc/resolv.conf lists no nameservers, or lists localhost or 127.0.0.1. They do not detect other loopback addresses, and they do not check /run/systemd/resolve/resolv.conf.

Pods also inherit the host's DNS search domains, in addition to the cluster's own search domains. Kubernetes sets the ndots:5 resolver option in pods, so a name with fewer than five dots is tried against every search domain before it is tried as written. On hosts with many search domains, this can make lookups slow, and some container images fail to resolve names that work on the host. For more information about resolving names from pods, see Pods cannot resolve names that resolve on the host in Troubleshoot Embedded Cluster.

Unix accounts for Kubernetes components

During installation, Embedded Cluster automatically creates the following Unix accounts that are required by internal Kubernetes components:

• etcd: Used by the Kubernetes etcd database, which stores cluster state.
• konnectivity-server: Used by the Konnectivity service, which facilitates secure communication between internal components.
• kube-apiserver: Used by the Kubernetes API server.
• kube-scheduler: Used by the Kubernetes scheduler to schedule workloads such as pods.

No action is required to create these roles. Removing them will make the cluster non-functional.

For more information about the internal Kubernetes components, see Kubernetes Components in the Kubernetes documentation.

For more information about the Konnectivity service, see Set up Konnectivity service in the Kubernetes documentation.

Firewall openings for online installations with Embedded Cluster

The domains for the services listed below need to be accessible from servers performing online installations. No outbound internet access is required for air gap installations.

For services hosted at domains owned by Replicated, the table includes a link to the list of IP addresses for the domain at replicatedhq/ips in GitHub. These are the addresses that Replicated services listen on, so they are the addresses that an online installation connects to.

That repository covers every Replicated service, so it lists more domains than any one installation needs. Use the table below to determine which of its entries apply to you.

note

Replicated does not publish the source IP addresses that its services use for outbound connections, such as when the Replicated proxy registry pulls an image from a vendor's registry, and does not commit to keeping those addresses stable. Do not use the addresses in replicatedhq/ips to build an allowlist for traffic originating from Replicated. Where access to a registry must be restricted, use registry credentials rather than a source IP allowlist.

Some of the domains listed below belong to public services that Replicated does not own, such as Docker Hub, GitHub and Amazon S3. Their IP addresses are not in replicatedhq/ips, so consult the provider's documentation for their ranges.

Domain	Description
proxy.replicated.com	Private Docker images are proxied through proxy.replicated.com. This domain is owned by Replicated, Inc., which is headquartered in Los Angeles, CA. For the range of IP addresses for proxy.replicated.com, see replicatedhq/ips in GitHub.
replicated.app	Upstream application YAML and metadata is pulled from replicated.app. The current running version of the application (if any), as well as a license ID and application ID to authenticate, are all sent to replicated.app. This domain is owned by Replicated, Inc., which is headquartered in Los Angeles, CA. For the range of IP addresses for replicated.app, see replicatedhq/ips in GitHub.
registry.replicated.com *	Some applications host private images in the Replicated registry at this domain. The on-prem docker client uses a license ID to authenticate to registry.replicated.com. This domain is owned by Replicated, Inc which is headquartered in Los Angeles, CA. For the range of IP addresses for registry.replicated.com, see replicatedhq/ips in GitHub.

* Required only if the application uses the Replicated private registry.

note

Embedded Cluster 2.13.0 and later disable the k0s update-prober feature, which checks updates.k0sproject.io for k0s updates, on installations that use Kubernetes 1.32 or later. The feature is disabled only on nodes that are installed or joined with one of these versions. Upgrading does not disable it on existing nodes. Nodes where the feature is still enabled call updates.k0sproject.io approximately every 30 minutes. Embedded Cluster does not use the update-prober feature, and blocking these calls in your firewall does not affect Replicated product functionality.

About firewalld configuration

When Firewalld is enabled in the installation environment, Embedded Cluster modifies the Firewalld config to allow traffic over the pod and service networks and to open the required ports on the host. No additional configuration is required.

The following rule is added to Firewalld:

<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <interface name="cali+"/>
  <interface name="tunl+"/>
  <interface name="vxlan-v6.calico"/>
  <interface name="vxlan.calico"/>
  <interface name="wg-v6.cali"/>
  <interface name="wireguard.cali"/>
  <source address="[pod-network-cidr]"/>
  <source address="[service-network-cidr]"/>
</zone>

The following ports are opened in the default zone:

Port	Protocol
6443	TCP
10250	TCP
9443	TCP
2380	TCP
4789	UDP

Security tools on the host

Endpoint security, firewall management, and application control tools, such as Illumio VEN or CrowdStrike Falcon, can interfere with Embedded Cluster. Replicated recommends disabling these tools on Embedded Cluster hosts. If disabling a tool is not an option in the installation environment, configure the tool to meet the requirements in the following table. Each requirement includes the reason that Embedded Cluster needs it. Configuration steps vary by product and version, so work with the tool's vendor to apply them.

Requirement	Why
Do not block, flush, replace, or reorder the KUBE-* and cali-* firewall rules, whether the host uses iptables or nftables. The usual approach is to exclude cluster nodes from the tool's host firewall management, or to use the tool's Kubernetes mode if it has one.	The cluster's networking components, kube-proxy and Calico, route all pod and service traffic through these rules. If a tool blocks the rules during installation, kube-proxy restarts repeatedly and the installation does not complete. If a tool removes the rules later, such as after the tool restarts, receives a policy update, or the host reboots, pods can no longer reach the Kubernetes API and the application stops working.
Allow traffic to and from the pod and service networks, and on the cali* and vxlan.calico network interfaces. For more information about the interfaces and networks, see About firewalld configuration in this document.	All traffic between pods, services, and the host uses these networks and interfaces. A default-deny policy drops this traffic, with the same result as removing the firewall rules.
Allow binaries to run from the Embedded Cluster executable paths. For more information about the paths, see Executable paths in this document.	Embedded Cluster runs Kubernetes and its networking components from these paths. If a tool blocks execution, installation or upgrade fails, or the node cannot start the cluster.
Do not quarantine or remove files in the Embedded Cluster executable paths or data directory. For more information about the executable paths, see Executable paths in this document. For more information about the data directory, see Data directory contents in this document. If the tool scans files, also exclude DATA_DIR/k0s/containerd from scanning, where DATA_DIR is the Embedded Cluster data directory.	Removing a cluster binary, a systemd unit, or a networking binary stops the node. Scanners that hold files open in container filesystems can also prevent containers from restarting.
Do not change the kernel modules and parameters that Embedded Cluster configures. For more information about the kernel modules and parameters, see Host configuration in this document.	Hardening tools that reset values such as net.ipv4.ip_forward or net.ipv4.conf.all.rp_filter drop pod traffic, even when no firewall rule changed.

Executable paths

Security tools that control which binaries can run must allow execution in the following paths, where DATA_DIR is the Embedded Cluster data directory:

• /usr/local/bin/k0s: The Kubernetes distribution that Embedded Cluster installs
• DATA_DIR/bin: Tools that Embedded Cluster uses to install and support the cluster
• DATA_DIR/k0s: Kubernetes components and the container runtime
• /opt/cni: Calico networking binaries

Data directory contents

Security tools often flag unfamiliar binaries and certificates in /var/lib. Knowing what each directory holds helps security teams write narrow exceptions instead of broad ones, which are easier to approve. The data directory contains the following:

• bin: Tools that Embedded Cluster uses to install and support the cluster
• charts: Helm charts for the components that Embedded Cluster installs
• images: Container image archives that Embedded Cluster uses during installation
• k0s: Kubernetes components and their data, including pod logs, the container runtime, and every container image and container filesystem on the node. This includes the application images. The pki subdirectory holds certificates that cluster components use only to communicate with each other.
• openebs-local: Persistent storage for the application
• seaweedfs: Storage for the internal registry in high availability air gap installations
• support: Files used for troubleshooting, such as support bundle specifications
• tmp: Temporary files that Embedded Cluster uses during installation and upgrade

Compliance scans on the host

Security teams often scan hosts against an operating system benchmark, such as a CIS Benchmark for the Linux distribution or a DISA STIG for the operating system. These benchmarks are written for general-purpose servers and do not account for Kubernetes. On an Embedded Cluster host, they flag settings and files that Kubernetes needs. An operating system benchmark still applies to the host. To assess the Kubernetes components that Embedded Cluster runs on the host, use a Kubernetes benchmark, such as the CIS Kubernetes Benchmark.

The following table lists findings that operating system benchmarks commonly report on Embedded Cluster hosts, and the reason for each. Do not change these settings or files to satisfy a scan. The usual approach is to record an exception for the finding or exclude the Embedded Cluster paths from the check.

Finding	Reason
Reverse path filtering (net.ipv4.conf.all.rp_filter and net.ipv4.conf.default.rp_filter) is not set to 1	Embedded Cluster sets these parameters to 2 (loose mode), which still drops packets from source addresses that have no route back. For more information about why Embedded Cluster does not use strict mode, see Host configuration in this document.
World-writable files, and world-writable directories without the sticky bit, under DATA_DIR/k0s/kubelet	The kubelet creates these as part of standard Kubernetes behavior. It creates each container's termination message file with mode 0666 so that a container running as any user can write it, and creates pod volumes with mode 0777 so that containers that do not run as root can use them. Changing these permissions can break the pods that use them.
Files with no owner or group	Files that containers create, such as files in persistent volumes, are owned by the user and group IDs that the container runs as. The host does not need accounts for these IDs.

In this table, DATA_DIR is the Embedded Cluster data directory.