TL;DR

On the machine where you build:

1
2
3

docker build -t test-app:v1.0.0 .
docker save -o test-app-v1.0.0.tar test-app:v1.0.0
rsync -v test-app-v1.0.0.tar node1:/tmp/

On every k3s node where the pod may land:

1
2

sudo k3s ctr images import /tmp/test-app-v1.0.0.tar
sudo k3s crictl images | grep test-app        # docker.io/library/test-app  v1.0.0

And in the Deployment, so kubelet does not try to pull it:

1
2
3
4
5

spec:
  containers:
  - name: test-app
    image: test-app:v1.0.0
    imagePullPolicy: Never

Why

k3s runs on containerd, not Docker. An image that exists in the Docker on the same machine is invisible to k3s, and the first time a pod asks for it kubelet tries to pull it from Docker Hub and fails with ErrImagePull. There are three ways out: a registry (the right one past a certain size), importing the tar on every node, or letting k3s import it by itself from a directory.

k3s ctr is containerd’s ctr already pointed at the k3s socket and the k8s.io namespace, which is where kubelet and crictl look. If you use the bare ctr binary you need --address /run/k3s/containerd/containerd.sock -n k8s.io.

Alternatives for more than one node

Images directory. Every .tar, .tar.gz, or .tar.zst you drop into /var/lib/rancher/k3s/agent/images/ is imported each time k3s starts (it is the documented mechanism for installs without Internet):

1
2
3

sudo mkdir -p /var/lib/rancher/k3s/agent/images/
sudo cp test-app-v1.0.0.tar /var/lib/rancher/k3s/agent/images/
sudo systemctl restart k3s        # or k3s-agent

Advantage over the manual import: if kubelet’s image garbage collector removes it, it comes back on the next start.

Embedded registry (Spegel). Since k3s 1.29 servers can start with --embedded-registry (or embedded-registry: true in /etc/rancher/k3s/config.yaml); each node publishes on the P2P network (port 5001) the images it already has and the others pull them from it instead of the Internet. It requires listing the registries to mirror in /etc/rancher/k3s/registries.yaml:

1
2
3

mirrors:
  docker.io:
  "*":

With that, you import the image on one node and the rest get it on their own, as long as the nodes can reach each other on 5001 and 6443.

Your own registry. A registry:2 or Harbor in the cluster, and docker push. It is the path once CI is involved.

Listing and cleaning

1
2
3
4

sudo k3s crictl images                    # what is on the node
sudo k3s crictl rmi --prune               # delete the ones no container uses
sudo k3s crictl rmi docker.io/library/test-app:v1.0.0
sudo k3s crictl rmi --all                 # everything; kubelet will pull back what it needs

kubelet also collects on its own: when the disk holding /var/lib/rancher/k3s/agent/containerd goes over 85 % it deletes unused images until it drops below 80 %. The thresholds are changed on the service:

1
2
3
4

# /etc/rancher/k3s/config.yaml
kubelet-arg:
  - image-gc-high-threshold=85
  - image-gc-low-threshold=80

If that disk runs short, moving the data-dir is another post.

Traps

  • Every node. A pod scheduled on a node without the image gives ErrImageNeverPull. Either import on all of them, or label the node and use nodeSelector, or move to the embedded registry.
  • Architecture. For ARM nodes (Raspberry Pi) the image must be linux/arm64: docker buildx build --platform linux/arm64.
  • latest hits the network. With the latest tag the default policy is Always and kubelet tries to pull it even if it exists. Use explicit tags and Never or IfNotPresent.
  • The full name of an imported image is docker.io/library/test-app:v1.0.0; in the manifest test-app:v1.0.0 resolves to the same thing.
  • podman: podman save --format docker-archive -o app.tar app:v1, the format ctr imports with no surprises.
  • A crictl rmi --prune also deletes what you imported if no container is using it at that moment. For what must survive, the agent/images/ directory.

References: Air-Gap Install and Embedded Registry Mirror in the k3s documentation.