Embedded Cluster installation requirements (Beta)

This topic lists the installation requirements for Replicated Embedded Cluster. Ensure that the installation environment meets these requirements before you attempt to install.

System requirements

• Linux operating system

• cgroups v2 (required for Kubernetes versions 1.35 and later)

• x86-64 architecture

• systemd

• At least 2GB of memory and 2 CPU cores

• The disk on the host must have a maximum P99 write latency of 10 ms. This supports etcd performance and stability. For more information about the disk write latency requirements for etcd, see Disks in Hardware recommendations and What does the etcd warning “failed to send out heartbeat on time” mean? in the etcd documentation.

• The user performing the installation must have root access to the machine, such as with sudo.

• The data directory used by Embedded Cluster must have 40Gi or more of total space and be less than 80% full. The data directory cannot be on a filesystem mounted with the noexec option, because Embedded Cluster runs binaries from it. By default, the data directory is /var/lib/APP_SLUG, where APP_SLUG is the unique slug of the application. The directory can be changed by passing the --data-dir flag with the Embedded Cluster install command. For more information, see install.

  For installations created with Embedded Cluster 3.17.0-beta.1 and later, pod logs default to <data-dir>/k0s/pod-logs and count toward data directory disk usage. Installations created with earlier versions or migrated from Embedded Cluster v2 retain their existing pod log directory, which defaults to /var/log/pods.

  Clusters restored using Embedded Cluster 3.17.0-beta.1 or later default to <data-dir>/k0s/pod-logs, even if the cluster was originally installed with an earlier version. An explicit podLogsDir setting in the worker profile is preserved during restore.

  Keep pod logs on the same filesystem as the data directory and kubelet root directory. Host preflight checks warn if the filesystems differ or if the paths cannot be resolved or inspected. Kubernetes might not account for logs on a different filesystem during disk-pressure eviction.

  For existing installations using /var/log/pods, this warning is expected during upgrades if the data directory is on a separate filesystem. The warning does not block the upgrade. Embedded Cluster does not move pod logs for existing installations.

  Note that in addition to the primary data directory, Embedded Cluster creates directories and files in the following locations:

  • /etc/cni
  • /etc/k0s
  • /opt/cni
  • /opt/containerd
  • /run/calico
  • /run/containerd
  • /run/k0s
  • /sys/fs/cgroup/kubepods
  • /sys/fs/cgroup/system.slice/containerd.service
  • /sys/fs/cgroup/system.slice/k0scontroller.service
  • /usr/libexec/k0s
  • /var/lib/calico
  • /var/lib/cni
  • /var/lib/containers
  • /var/lib/ec
  • /var/lib/kubelet
  • /var/log/calico
  • /var/log/containers
  • /var/log/APP_SLUG, where APP_SLUG is the unique slug for the application
  • /var/log/pods (the default for installations created before Embedded Cluster 3.17.0-beta.1 or migrated from Embedded Cluster v2; also used when podLogsDir is explicitly empty)
  • /usr/local/bin/k0s

• (Online installations only) Access to replicated.app and proxy.replicated.com or your custom domain for each

• Embedded Cluster is based on k0s, so all k0s system requirements and external runtime dependencies apply. See System requirements and External runtime dependencies in the k0s documentation.

Host configuration

Embedded Cluster configures most of the required kernel settings itself at the start of installation, before the host preflight checks run. The preflight checks then verify the result and report anything that could not be applied.

This means you do not need to set these values in advance. This section describes what Embedded Cluster changes on the host, which is often needed for security review, and what the host must provide for those changes to succeed.

Kernel settings that Embedded Cluster configures

Every change below is best effort. If one cannot be applied, installation continues and the host preflight checks report the problem.

Kernel modules. Loaded immediately, and written to /etc/modules-load.d/99-embedded-cluster.conf so that they persist across reboots:

overlay, ip_tables, nf_tables, nft_compat, br_netfilter, and nf_conntrack

Network parameters. Written to /etc/sysctl.d/99-embedded-cluster.conf and applied:

Parameter	Value
net.ipv4.ip_forward	1
net.ipv4.conf.all.forwarding	1
net.ipv4.conf.default.forwarding	1
net.ipv6.conf.all.forwarding	1
net.ipv6.conf.default.forwarding	1
net.bridge.bridge-nf-call-iptables	1
net.bridge.bridge-nf-call-ip6tables	1
net.ipv4.conf.all.arp_filter	0
net.ipv4.conf.default.arp_filter	0
net.ipv4.conf.all.arp_ignore	0
net.ipv4.conf.default.arp_ignore	0
net.ipv4.conf.all.rp_filter	2
net.ipv4.conf.default.rp_filter	2

Reverse path filtering is set to loose mode. Strict mode, which is the value 1, drops pod traffic.

Resource limits. Written to /etc/sysctl.d/99-dynamic-embedded-cluster.conf, and only for values that are currently lower on the host:

Parameter	Minimum value
fs.inotify.max_user_instances	1024
fs.inotify.max_user_watches	65536

What the host must provide

Embedded Cluster cannot supply the following, so the host must have them before you install:

• The modprobe command. If it is missing, Embedded Cluster skips kernel module setup entirely.
• The sysctl command. If it is missing, Embedded Cluster writes the network parameter file but cannot apply it until the host reboots, and does not set the inotify limits at all.
• The mount and umount commands.
• A kernel that provides the modules listed above. Embedded Cluster can load them, but cannot supply them on a kernel that was built without them.
• The following cgroup controllers: cpu, cpuacct, cpuset, memory, devices, freezer, and pids.
• A hostname of 54 characters or fewer. Internal component labels are derived from the node name, and a longer hostname exceeds the Kubernetes 63-character label limit, which prevents the cluster from starting.

Hardened hosts

On hosts that enforce their own kernel parameter policy, through configuration management or a file in /etc/sysctl.d that loads after the Embedded Cluster file, that policy can override these values after Embedded Cluster applies them. The host preflight checks report the resulting mismatch. Allow the values above in the host's policy rather than setting them by hand.

Port requirements

This section lists the ports used by Embedded Cluster. These ports must be open and available for both single- and multi-node installations.

Ports used by local processes

The following ports must be open and available for use by local processes running on the same node. It is not necessary to create firewall openings for these ports.

• 2379/TCP
• 7443/TCP
• 9099/TCP
• 10248/TCP
• 10257/TCP
• 10259/TCP

Ports required for bidirectional communication between nodes

The following ports are used for bidirectional communication between nodes.

For multi-node installations, create firewall openings between nodes for these ports.

For single-node installations, ensure that there are no other processes using these ports. Although there is no communication between nodes in single-node installations, these ports are still required.

• 2380/TCP
• 4789/UDP
• 6443/TCP
• 9091/TCP
• 9443/TCP
• 10249/TCP
• 10250/TCP
• 10256/TCP

Embedded Cluster console and API port

The Embedded Cluster daemon serves the admin console and its HTTP API on port 30000/TCP on every node. Users open the admin console in a browser on this port. Nodes use it to join the cluster and to receive streamed upgrades, and external automation uses it for headless installs and programmatic node joins.

Create a firewall opening for port 30000/TCP so that users can reach the admin console. For multi-node installations, also open this port between nodes.

If port 30000 is occupied, you can select a different port with --api-port during installation. The port cannot be changed after installation. For more information, see install and External API.

In Embedded Cluster versions earlier than 3.17.0-beta.1, the installer UI and admin console use port 30080/TCP and the API uses port 30081/TCP. When you upgrade a cluster that uses the default ports to 3.17.0-beta.1 or later, the console and API move to port 30000, so open port 30000/TCP before you upgrade. A cluster installed with a custom --api-port keeps that port for both the console and the API.

DNS on the host

Embedded Cluster runs CoreDNS to resolve names inside the cluster. CoreDNS forwards names that it cannot resolve itself, such as the host name of an external database, to the nameservers configured on the host. The host must meet the following requirements:

• /etc/resolv.conf lists at least one nameserver.

• /etc/resolv.conf does not list a loopback address, such as localhost, 127.0.0.1, or 127.0.1.1, as a nameserver. Inside the cluster, a loopback address points CoreDNS back to itself, so queries loop instead of reaching a real nameserver.

  Hosts that use systemd-resolved can list its local stub resolver, 127.0.0.53, in /etc/resolv.conf, as long as it is the only nameserver listed. In that case, Embedded Cluster uses the upstream nameservers from /run/systemd/resolve/resolv.conf instead, so that file must list at least one nameserver. If /etc/resolv.conf lists 127.0.0.53 alongside other nameservers, Embedded Cluster uses /etc/resolv.conf as it is, and CoreDNS cannot reach 127.0.0.53.

The host preflight checks fail if /etc/resolv.conf lists no nameservers, or lists localhost or 127.0.0.1. They do not detect other loopback addresses, and they do not check /run/systemd/resolve/resolv.conf.

Pods also inherit the host's DNS search domains, in addition to the cluster's own search domains. Kubernetes sets the ndots:5 resolver option in pods, so a name with fewer than five dots is tried against every search domain before it is tried as written. On hosts with many search domains, this can make lookups slow, and some container images fail to resolve names that work on the host. For more information about resolving names from pods, see Pods cannot resolve names that resolve on the host in Troubleshoot Embedded Cluster.

Unix accounts for Kubernetes components

During installation, Embedded Cluster automatically creates the following Unix accounts, which internal Kubernetes components require:

• etcd: Used by the Kubernetes etcd database, which stores cluster state.
• konnectivity-server: Used by the Konnectivity service, which facilitates secure communication between internal components.
• kube-apiserver: Used by the Kubernetes API server.
• kube-scheduler: Used by the Kubernetes scheduler to schedule workloads such as pods.

You do not need to take any action to create these roles. Removing them will make the cluster non-functional.

For more information about the internal Kubernetes components, see Kubernetes Components in the Kubernetes documentation.

For more information about the Konnectivity service, see Set up Konnectivity service in the Kubernetes documentation.

Firewall openings for online installations with Embedded Cluster

The domains for the services listed below need to be accessible from servers performing online installations. No outbound internet access is required for air gap installations.

For services hosted at domains owned by Replicated, the table includes a link to the list of IP addresses for the domain at replicatedhq/ips in GitHub. These are the addresses that Replicated services listen on, so they are the addresses that an online installation connects to.

That repository covers every Replicated service, so it lists more domains than any one installation needs. Use the table below to determine which of its entries apply to you.

note

Replicated does not publish the source IP addresses that its services use for outbound connections, such as when the Replicated proxy registry pulls an image from a vendor's registry, and does not commit to keeping those addresses stable. Do not use the addresses in replicatedhq/ips to build an allowlist for traffic originating from Replicated. Where access to a registry must be restricted, use registry credentials rather than a source IP allowlist.

Some of the domains listed below belong to public services that Replicated does not own, such as Docker Hub, GitHub and Amazon S3. Their IP addresses are not in replicatedhq/ips, so consult the provider's documentation for their ranges.

Domain	Description
proxy.replicated.com	Private Docker images are proxied through proxy.replicated.com. This domain is owned by Replicated, Inc., which is headquartered in Los Angeles, CA. For the range of IP addresses for proxy.replicated.com, see replicatedhq/ips in GitHub.
replicated.app	Upstream application YAML and metadata is pulled from replicated.app. The current running version of the application (if any), as well as a license ID and application ID to authenticate, are all sent to replicated.app. This domain is owned by Replicated, Inc., which is headquartered in Los Angeles, CA. For the range of IP addresses for replicated.app, see replicatedhq/ips in GitHub.
registry.replicated.com *	Some applications host private images in the Replicated registry at this domain. The on-prem docker client uses a license ID to authenticate to registry.replicated.com. This domain is owned by Replicated, Inc which is headquartered in Los Angeles, CA. For the range of IP addresses for registry.replicated.com, see replicatedhq/ips in GitHub.

* Required only if the application uses the Replicated private registry.

note

Embedded Cluster v3 disables the update-prober feature, which checks updates.k0sproject.io for updates, on every node that it installs or joins, so these nodes do not call updates.k0sproject.io. In installations upgraded from Embedded Cluster v2, nodes that existed before the upgrade keep their v2 setting, and might call updates.k0sproject.io approximately every 30 minutes. Embedded Cluster does not use the update-prober feature, and blocking these calls in your firewall does not affect Replicated product functionality.

About firewalld configuration

When Firewalld runs in the installation environment, Embedded Cluster modifies the Firewalld config to allow traffic over the pod and service networks. It also opens ports 2380, 6443, 9443, and 10250/TCP and port 4789/UDP. Embedded Cluster does not open the console and API port (30000/TCP by default) or ports 9091, 10249, and 10256/TCP. Create firewall openings for those ports yourself. See Port requirements.

Embedded Cluster adds the following rule to Firewalld:

<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <interface name="cali+"/>
  <interface name="tunl+"/>
  <interface name="vxlan-v6.calico"/>
  <interface name="vxlan.calico"/>
  <interface name="wg-v6.cali"/>
  <interface name="wireguard.cali"/>
  <source address="[pod-network-cidr]"/>
  <source address="[service-network-cidr]"/>
</zone>

Embedded Cluster opens the following ports in the default zone:

Port	Protocol
6443	TCP
10250	TCP
9443	TCP
2380	TCP
4789	UDP

Security tools on the host

Endpoint security, firewall management, and application control tools, such as Illumio VEN or CrowdStrike Falcon, can interfere with Embedded Cluster. Replicated recommends disabling these tools on Embedded Cluster hosts. If disabling a tool is not an option in the installation environment, configure the tool to meet the requirements in the following table. Each requirement includes the reason that Embedded Cluster needs it. Configuration steps vary by product and version, so work with the tool's vendor to apply them.

Requirement	Why
Do not block, flush, replace, or reorder the KUBE-* and cali-* firewall rules, whether the host uses iptables or nftables. The usual approach is to exclude cluster nodes from the tool's host firewall management, or to use the tool's Kubernetes mode if it has one.	The cluster's networking components, kube-proxy and Calico, route all pod and service traffic through these rules. If a tool blocks the rules during installation, kube-proxy restarts repeatedly and the installation does not complete. If a tool removes the rules later, such as after the tool restarts, receives a policy update, or the host reboots, pods can no longer reach the Kubernetes API and the application stops working.
Allow traffic to and from the pod and service networks, and on the cali* and vxlan.calico network interfaces. For more information about the interfaces and networks, see About firewalld configuration in this document.	All traffic between pods, services, and the host uses these networks and interfaces. A default-deny policy drops this traffic, with the same result as removing the firewall rules.
Allow binaries to run from the Embedded Cluster executable paths. For more information about the paths, see Executable paths in this document.	Embedded Cluster runs Kubernetes and its networking components from these paths. If a tool blocks execution, installation or upgrade fails, or the node cannot start the cluster.
Do not quarantine or remove files in the Embedded Cluster executable paths or data directory. For more information about the executable paths, see Executable paths in this document. For more information about the data directory, see Data directory contents in this document. If the tool scans files, also exclude DATA_DIR/k0s/containerd from scanning, where DATA_DIR is the Embedded Cluster data directory.	Removing a cluster binary, a systemd unit, or a networking binary stops the node. Scanners that hold files open in container filesystems can also prevent containers from restarting.
Do not change the kernel modules and parameters that Embedded Cluster configures. For more information about the kernel modules and parameters, see Host configuration in this document.	Hardening tools that reset values such as net.ipv4.ip_forward or net.ipv4.conf.all.rp_filter drop pod traffic, even when no firewall rule changed.

Executable paths

Security tools that control which binaries can run must allow execution in the following paths, where DATA_DIR is the Embedded Cluster data directory:

• /usr/local/bin: The Embedded Cluster CLI and its services
• DATA_DIR/bin: Kubernetes and the tools that Embedded Cluster uses to install and support the cluster
• DATA_DIR/k0s: Kubernetes components and the container runtime
• DATA_DIR/tmp: Scripts that Embedded Cluster runs during installation and upgrade
• /opt/cni: Calico networking binaries

Before it installs or upgrades, Embedded Cluster checks that the first four paths allow execution. If a path does not, Embedded Cluster stops with an error that names the blocked path.

Data directory contents

Security tools often flag unfamiliar binaries and certificates in /var/lib. Knowing what each directory holds helps security teams write narrow exceptions instead of broad ones, which are easier to approve. The data directory contains the following:

• bin: Kubernetes and the tools that Embedded Cluster uses to install and support the cluster
• k0s: Kubernetes components and their data, including the container runtime and every container image and container filesystem on the node. This includes the application images. The pki subdirectory holds certificates that cluster components use only to communicate with each other.
• openebs-local: Persistent storage for the application
• tmp: Temporary files that Embedded Cluster uses during installation and upgrade

Compliance scans on the host

Security teams often scan hosts against an operating system benchmark, such as a CIS Benchmark for the Linux distribution or a DISA STIG for the operating system. These benchmarks are written for general-purpose servers and do not account for Kubernetes. On an Embedded Cluster host, they flag settings and files that Kubernetes needs. An operating system benchmark still applies to the host. To assess the Kubernetes components that Embedded Cluster runs on the host, use a Kubernetes benchmark, such as the CIS Kubernetes Benchmark.

The following table lists findings that operating system benchmarks commonly report on Embedded Cluster hosts, and the reason for each. Do not change these settings or files to satisfy a scan. The usual approach is to record an exception for the finding or exclude the Embedded Cluster paths from the check.

Finding	Reason
Reverse path filtering (net.ipv4.conf.all.rp_filter and net.ipv4.conf.default.rp_filter) is not set to 1	Embedded Cluster sets these parameters to 2 (loose mode), which still drops packets from source addresses that have no route back. For more information about why Embedded Cluster does not use strict mode, see Host configuration in this document.
World-writable files, and world-writable directories without the sticky bit, under DATA_DIR/k0s/kubelet	The kubelet creates these as part of standard Kubernetes behavior. It creates each container's termination message file with mode 0666 so that a container running as any user can write it, and creates pod volumes with mode 0777 so that containers that do not run as root can use them. Changing these permissions can break the pods that use them.
Files with no owner or group	Files that containers create, such as files in persistent volumes, are owned by the user and group IDs that the container runs as. The host does not need accounts for these IDs.

In this table, DATA_DIR is the Embedded Cluster data directory.

Embedded Cluster v3 also links /var/lib/ec/kubelet to DATA_DIR/k0s/kubelet, so scanners might report the same kubelet files under /var/lib/ec/kubelet.