容器运行时:containerd

Demon.Lee 2022年12月07日 1,479次浏览

本文演示环境:

demon@ubuntu2204:~$ hostnamectl 
Static hostname: ubuntu2204
      Chassis: vm
Virtualization: vmware
Operating System: Ubuntu 22.04.1 LTS              
       Kernel: Linux 5.15.0-57-generic
 Architecture: arm64
demon@ubuntu2204:~$

笔者曾根据周志明老师《凤凰架构》一书中的内容,对虚拟化技术与文件隔离进行了简单的总结,又在《容器的基本操作和实现原理》一文中梳理了 Docker 容器的基本实现逻辑。但随着容器编排领域的角逐,Kubernetes 牢牢把控住了云原生体系的头把交椅,而 Docker 则逐渐被边缘化。

以下内容来自 Kubernetes 官方文档

自 1.24 版起,Dockershim 已从 Kubernetes 项目中移除。阅读 Dockershim 移除的常见问题了解更多详情。

v1.24 之前的 Kubernetes 版本直接集成了 Docker Engine 的一个组件,名为 dockershim。 这种特殊的直接整合不再是 Kubernetes 的一部分 (这次删除被作为 v1.20 发行版本的一部分宣布)。

你可以阅读检查 Dockershim 移除是否会影响你以了解此删除可能会如何影响你。 要了解如何使用 dockershim 进行迁移, 请参阅从 dockershim 迁移

也就是说,自 Kubernetes v1.24 起,默认的容器运行时不再是 Docker Engine,而换成了 containerd。如果不了解其中的细节,可能会引起恐慌,因为我们很多代码都是用 docker 跑起来,突然又来了一个新东西,是不是将来又得再搞一遍?

其实没必要,这一节,笔者就来简单聊聊 containerd,并结合具体的安装与使用,比较它和 docker 之间的差异。

容器运行时

从 Kubernetes 官网可以看到,其中列出了多个不同的容器运行时:

那么,我们脑中闪现的第一个问题可能是:什么是容器运行时?

这就不得不提 OCI (Open Container Initiative,开放容器标准) 规范。它由 Docker 倡导和提议,联合 CoreOS 等多家厂商,于 2015 年 6 月 22 日发起。OCI 主要包括三个部分,其中一个就是运行时标准:

  • runtime-spec:容器运行时标准,定义了容器运行的配置,环境和生命周期。即如何运行一个容器,如何管理容器的状态和生命周期,如何使用操作系统的底层特性(namespace,cgroup,pivot_root 等);
  • image-spec:容器镜像标准,定义了镜像的格式,配置(包括应用程序的参数,环境信息等),依赖的元数据格式等,简单来说就是对镜像的静态描述;
  • distribution-spec:镜像分发标准,即规定了镜像上传和下载的网络交互过程。

关于这些标准的更多细节,可以查阅官方文档进一步了解。

对于容器运行时,笔者参考这篇文章,梳理了容器启动后的执行步骤:

​ 1)拉取镜像:如果本地没有镜像,从镜像仓库拉取;

​ 2)镜像解压缩:镜像被解压缩到一个 COW(copy-on-write,写时复制)的文件系统上,所有容器层相互堆叠,最终合并成一个文件系统;

​ 3)为容器准备一个挂载点;

​ 4)设置容器:从镜像中获取元数据信息,并对容器进行设置,保证容器能按照期望的方式正常启动,设置过程中还包括来自用户的输入,比如 CMD,ENTRYPOINT 或程序启动的入参等;

​ 5)设置 namespaces:内核为容器设置相关隔离,如文件系统,进程,网络,IPC 等;

​ 6)设置 cgroups:内核为容器设置资源使用限制,如 CPU,memory 等;

​ 7)通过系统调用启动容器;

​ 8)设置 SELinux/AppArmor,即为容器设置相关安全策略。

到这里,相信读者对容器运行时已经有了一个初步的概念,简单来说就是容器的全生命周期管理,包括创建容器,运行,停止,删除等。但在介绍 containerd 之前,我们先来看看它的先辈们: LXC 和 Docker。

LXC

说起容器,一定会提到 Docker,但在这之前,其实早就有了类似的容器技术,这就是 LXC(LinuX Containers,Linux 容器)。

我们知道,2008 年发布的 Linux 2.24 内核首次引入了 Cgroups 技术,而在同一时间,Linux 就发布了 LXC。


查看 LXC 在维基百科上的定义:

Linux Containers (LXC) is an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a control host using a single Linux kernel.

The Linux kernel provides the cgroups functionality that allows limitation and prioritization of resources (CPU, memory, block I/O, network, etc.) without the need for starting any virtual machines, and also the namespace isolation functionality that allows complete isolation of an application’s view of the operating environment, including process trees, networking, user IDs and mounted file systems.

LXC combines the kernel’s cgroups and support for isolated namespaces to provide an isolated environment for applications. Early versions of Docker used LXC as the container execution driver, though LXC was made optional in v0.9 and support was dropped in Docker v1.10.

从定义中,我们了解到,LXC 是一个操作系统级的虚拟化方式,对 Cgroups 和 Namespaces 等 Linux 内核特性进行封装,然后提供一个统一的接口,降低用户使用容器技术的门槛。

但为啥 LXC 没有火,Docker 确火了呢?它们在技术层面没啥差别啊。

技术层面确实如此,但设计理念却完全不同:LXC 是封装系统的轻量级虚拟机[1],而 Docker 则是封装应用。

也就是说,LXC 是先装系统,再安装软件,如果想变更软件,则需要对安装的系统进行重新配置。比如部署 LAMP(Linux,Apache,MySQL,PHP)应用,需要先将基础环境安装好,再在里面把软件部署好,将其作为一个整体交付。如果我想将 MySQL 版本从 5.6 升级到 8.0,就得重新操作一遍,这就是封装系统的意思。

但 Docker 不一样,它依靠统一的内核,从应用的角度出发,每个容器默认只跑一个程序。如果是部署 LAMP,那就是 4 个容器,然后再将其组合起来。这对开发人员来说简直不能再友好了,不同的软件使用不同的镜像版本。这就是为什么 Docker 在起始阶段没有啥技术创新,却依靠一个好点子引爆一个时代的原因。

其实在 Docker 官网,Docker 创始人 Solomon Hykes 已经给出了历史选择 Docker 而不是 LXC 的原因:

  • Portable deployment across machines
  • Application-centric
  • Automatic build
  • Versioning
  • Component re-use
  • Sharing
  • Tool ecosystem

回到 LXC 的定义,我们还能发现一点,Docker 在 v1.10 版本之前是直接利用 LXC 来实现容器隔离的。但到了 2014 年,Docker 公司利用 Go 语言开发了新的底层驱动 libcontainer,从而越过 LXC 直接操控 Namespaces 和 Cgroups。如此一来,Docker 便能直接与系统内核打交道,LXC 也就在 Docker 的崛起中被其淘汰了。

containerd

上面简单梳理了 LXC 和 Docker 的故事,那这个 containerd 又是从哪冒出来的呢?

用一个不太准确的说法,其实它还是 Docker 的后代。

在说 containerd 的历史之前,我们先来看看 Docker 的架构。下图来自 Docker 官网,并且官网还给出了如下说明:

Docker uses a client-server architecture. The Docker client talks to the Docker daemon, which does the heavy lifting of building, running, and distributing your Docker containers. The Docker client and daemon can run on the same system, or you can connect a Docker client to a remote Docker daemon. The Docker client and daemon communicate using a REST API, over UNIX sockets or a network interface. Another Docker client is Docker Compose, that lets you work with applications consisting of a set of containers.


结合架构图可以看到,Docker 是典型的 C/S 架构,Client 和 Server 通过 REST API 交互。相对于客户端,服务端(即 Docker daemon)是主要的幕后功臣,由它完成构建、运行和分发容器的大量工作。从整体上来看,Docker 由 Docker Client,Docker daemon,Docker Registry,Docker Container 等多个子系统,以及 Graph,Driver,libcontainer 等多个模块共同构成。

前面提到,2014 年 Docker 使用 Go 语言开发了 libcontainer,并将底层的 LXC 替换掉。接着在 2015 年,Docker 又带头搞出了 OCI 标准。为了推动 OCI 标准的落地,Docker 进一步向前演进自身的架构:

1)先将 libcontainer 独立出来,重构成了 runc 项目,并将其捐献给了 Linux 基金会。

runc is a CLI tool for spawning and running containers on Linux according to the OCI specification.

Please note that runc is a low level tool not designed with an end user in mind. It is mostly employed by other higher level container software.

Therefore, unless there is some specific use case that prevents the use of tools like Docker or Podman, it is not recommended to use runc directly.

也就是说,现在由 runc 来负责底层容器的生成和运行,即直接操控 Namespaces 和 Cgroups 等内核特性。官方还特地说明,runc 是一个底层工具,不适合终端用户直接操作,最好配合高层次的容器软件使用。

那目前与 runc 配合使用的主要是谁呢?没错,就是 containerd。

2)Docker 继续重构 Docker daemon 子系统,将其中与容器运行时交互的部分抽象为 containerd 项目,并将其捐献给了 CNCF。

containerd is an industry-standard container runtime with an emphasis on simplicity, robustness and portability. It is available as a daemon for Linux and Windows, which can manage the complete container lifecycle of its host system: image transfer and storage, container execution and supervision, low-level storage and network attachments, etc.

containerd is designed to be embedded into a larger system, rather than being used directly by developers or end-users.

containerd 负责主机上容器的生命周期管理:包括镜像的传输和存储,容器的执行、监管、日志、构建、网络等功能。同样,它不也适合开发人员直接使用,而是嵌入到更大的系统中。

实际上,containerd 运行后,内部会创建一个 containerd-shim 进程,与 runc 搭配使用。下面是一个示例,启动一个 docker-registry-2.8.1 的容器后,再查询 containerd 相关的进程,就会发现 containerd-shim 的身影:

demon@ubuntu-2204:~$ ps aux|grep -v grep|grep container
root         833  0.0  1.2 1261920 37328 ?       Ssl  09:40   0:44 /usr/bin/containerd
root         990  0.0  1.9 1502972 58396 ?       Ssl  09:40   0:27 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
demon@ubuntu-2204:~$ 
demon@ubuntu-2204:~$ docker start docker-registry-2.8.1 
docker-registry-2.8.1
demon@ubuntu-2204:~$ 
demon@ubuntu-2204:~$ ps aux|grep -v grep|grep container
root         833  0.0  1.2 1261920 37552 ?       Ssl  09:40   0:44 /usr/bin/containerd

root         990  0.0  1.9 1502972 58684 ?       Ssl  09:40   0:27 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

root       20779  0.0  0.1 1148224 3100 ?        Sl   22:58   0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 5000 -container-ip 172.17.0.2 -container-port 5000

root       20786  0.0  0.1 1148224 3048 ?        Sl   22:58   0:00 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 5000 -container-ip 172.17.0.2 -container-port 5000

root       20804  0.5  0.2 711612  7344 ?        Sl   22:58   0:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 0d0266bf405938889084f282cdd22599ef8fb3c2b23a28981ebbcfed8b8b4279 -address /run/containerd/containerd.sock
demon@ubuntu-2204:~$

到这里,我们来回顾一下 Docker,containerd,runc 三者之间的关系:

来源:icyfenix.cn

从图中我们可以看到,Docker 也是靠 containerd 跑起来的,这就说明 Docker 还在更高的抽象层。

那么,Kubernetes 是如何做到将底层依赖的 Docker 干掉,换成 containerd 的?这又是另外一个 Kubernetes 与 Docker 相爱相杀的故事了,笔者将在后续的文章中进行梳理。不过,从这个结果,我们也可以反推:Kubernetes 不需要 Docker 高层的能力,只需要使用 containerd 管理容器就够了。

好,了解了 containerd 之后,我们再来看看如何使用它。

containerd 的安装与使用

再次提醒: 本机为 Linux Debian Arm64 架构,安装时请选择对应版本的安装包。

安装 containerd 及 runc 等

其实官方已经给出了详细的安装文档,但笔者还是遇到一些问题,故将操作步骤贴到这里,供大家参考。

Step 1: 安装 containerd

1)下载 containerd 并安装到指定路径

wget -c https://github.com/containerd/containerd/releases/download/v1.6.10/containerd-1.6.10-linux-arm64.tar.gz

sudo tar Cxzvf /usr/local containerd-1.6.10-linux-arm64.tar.gz

2)通过 systemd 启动 containerd

wget -c https://raw.githubusercontent.com/containerd/containerd/main/containerd.service

sudo mkdir -p /usr/local/lib/systemd/system/

sudo cp containerd.service /usr/local/lib/systemd/system/

sudo chmod 755 /usr/local/lib/systemd/system/containerd.service

sudo systemctl daemon-reload

sudo systemctl enable --now containerd

3)配置 containerd

通过命令获取默认配置内容:

sudo mkdir -p /etc/containerd/
containerd config default | sudo tee /etc/containerd/config.toml

修改 config.toml 中的 sandbox_image(可选),systemd_cgroupSystemdCgroup 等属性:

 # 将 sandbox_image = "registry.k8s.io/pause:3.6" 调整为
 sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8"
 
 # 将 [plugins."io.containerd.grpc.v1.cri"] 下的 systemd_cgroup = false 注释掉
 # systemd_cgroup = false
 
 # 将 [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options] 下的 SystemdCgroup = false 调整为 true
 SystemdCgroup = true

这里 sandbox_image 版本变成 3.8 是根据 Kubernetes 的版本来的(如果不是安装 Kubernetes,可以不用修改):

demon@ubuntu2204-master:~/Downloads$ sudo kubeadm config images list --kubernetes-version v1.25.3
registry.k8s.io/kube-apiserver:v1.25.3
registry.k8s.io/kube-controller-manager:v1.25.3
registry.k8s.io/kube-scheduler:v1.25.3
registry.k8s.io/kube-proxy:v1.25.3
registry.k8s.io/pause:3.8
registry.k8s.io/etcd:3.5.4-0
registry.k8s.io/coredns/coredns:v1.9.3
demon@ubuntu2204-master:~/Downloads$

另外,如果从本地私有仓库拉取镜像,可能会遇到如下错误:Unable to pull image from insecure registry, http: server gave HTTP response to HTTPS client,为此我们需要对这个私有仓库(比如这里的 http://192.168.10.127:5000)进行安全设置:

# 在 [plugins."io.containerd.grpc.v1.cri".registry] 下对 mirrors 和 mirrors 做如下配置
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.10.127:5000"]
          endpoint = ["https://192.168.10.127:5000", "http://192.168.10.127:5000"]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.10.127:5000".tls]
          insecure_skip_verify = true

修改后的版本样例:

disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []
root = "/var/lib/containerd"
state = "/run/containerd"
temp = ""
version = 2

[cgroup]
  path = ""

[debug]
  address = ""
  format = ""
  gid = 0
  level = ""
  uid = 0

[grpc]
  address = "/run/containerd/containerd.sock"
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
  tcp_address = ""
  tcp_tls_ca = ""
  tcp_tls_cert = ""
  tcp_tls_key = ""
  uid = 0

[metrics]
  address = ""
  grpc_histogram = false

[plugins]

  [plugins."io.containerd.gc.v1.scheduler"]
    deletion_threshold = 0
    mutation_threshold = 100
    pause_threshold = 0.02
    schedule_delay = "0s"
    startup_delay = "100ms"

  [plugins."io.containerd.grpc.v1.cri"]
    device_ownership_from_security_context = false
    disable_apparmor = false
    disable_cgroup = false
    disable_hugetlb_controller = true
    disable_proc_mount = false
    disable_tcp_service = true
    enable_selinux = false
    enable_tls_streaming = false
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    ignore_image_defined_volumes = false
    max_concurrent_downloads = 3
    max_container_log_line_size = 16384
    netns_mounts_under_state_dir = false
    restrict_oom_score_adj = false
    sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8"
    selinux_category_range = 1024
    stats_collect_period = 10
    stream_idle_timeout = "4h0m0s"
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    #systemd_cgroup = false
    tolerate_missing_hugetlb_controller = true
    unset_seccomp_profile = ""

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/opt/cni/bin"
      conf_dir = "/etc/cni/net.d"
      conf_template = ""
      ip_pref = ""
      max_conf_num = 1

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = true
      discard_unpacked_layers = false
      ignore_rdt_not_enabled_errors = false
      no_pivot = false
      snapshotter = "overlayfs"

      [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.default_runtime.options]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = ""
          cni_conf_dir = ""
          cni_max_conf_num = 0
          container_annotations = []
          pod_annotations = []
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_path = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = true

      [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime]
        base_runtime_spec = ""
        cni_conf_dir = ""
        cni_max_conf_num = 0
        container_annotations = []
        pod_annotations = []
        privileged_without_host_devices = false
        runtime_engine = ""
        runtime_path = ""
        runtime_root = ""
        runtime_type = ""

        [plugins."io.containerd.grpc.v1.cri".containerd.untrusted_workload_runtime.options]

    [plugins."io.containerd.grpc.v1.cri".image_decryption]
      key_model = "node"

    [plugins."io.containerd.grpc.v1.cri".registry]
      config_path = ""

      [plugins."io.containerd.grpc.v1.cri".registry.auths]

      [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
        [plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.10.127:5000"]
          endpoint = ["https://192.168.10.127:5000", "http://192.168.10.127:5000"]

      [plugins."io.containerd.grpc.v1.cri".registry.configs]
        [plugins."io.containerd.grpc.v1.cri".registry.configs."192.168.10.127:5000".tls]
          insecure_skip_verify = true

      [plugins."io.containerd.grpc.v1.cri".registry.headers]

    [plugins."io.containerd.grpc.v1.cri".x509_key_pair_streaming]
      tls_cert_file = ""
      tls_key_file = ""

  [plugins."io.containerd.internal.v1.opt"]
    path = "/opt/containerd"

  [plugins."io.containerd.internal.v1.restart"]
    interval = "10s"

  [plugins."io.containerd.internal.v1.tracing"]
    sampling_ratio = 1.0
    service_name = "containerd"

  [plugins."io.containerd.metadata.v1.bolt"]
    content_sharing_policy = "shared"

  [plugins."io.containerd.monitor.v1.cgroups"]
    no_prometheus = false

  [plugins."io.containerd.runtime.v1.linux"]
    no_shim = false
    runtime = "runc"
    runtime_root = ""
    shim = "containerd-shim"
    shim_debug = false

  [plugins."io.containerd.runtime.v2.task"]
    platforms = ["linux/arm64/v8"]
    sched_core = false

  [plugins."io.containerd.service.v1.diff-service"]
    default = ["walking"]

  [plugins."io.containerd.service.v1.tasks-service"]
    rdt_config_file = ""

  [plugins."io.containerd.snapshotter.v1.aufs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.btrfs"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.devmapper"]
    async_remove = false
    base_image_size = ""
    discard_blocks = false
    fs_options = ""
    fs_type = ""
    pool_name = ""
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.native"]
    root_path = ""

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""
    upperdir_label = false

  [plugins."io.containerd.snapshotter.v1.zfs"]
    root_path = ""

  [plugins."io.containerd.tracing.processor.v1.otlp"]
    endpoint = ""
    insecure = false
    protocol = ""

[proxy_plugins]

[stream_processors]

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar"

  [stream_processors."io.containerd.ocicrypt.decoder.v1.tar.gzip"]
    accepts = ["application/vnd.oci.image.layer.v1.tar+gzip+encrypted"]
    args = ["--decryption-keys-path", "/etc/containerd/ocicrypt/keys"]
    env = ["OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf"]
    path = "ctd-decoder"
    returns = "application/vnd.oci.image.layer.v1.tar+gzip"

[timeouts]
  "io.containerd.timeout.bolt.open" = "0s"
  "io.containerd.timeout.shim.cleanup" = "5s"
  "io.containerd.timeout.shim.load" = "5s"
  "io.containerd.timeout.shim.shutdown" = "3s"
  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

重启 containerd:

sudo systemctl restart containerd

Step 2: 安装 runc

wget -c https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.arm64

sudo install -m 755 runc.arm64 /usr/local/sbin/runc

Step 3: 安装 CNI plugins

sudo mkdir -p /opt/cni/bin

sudo tar Cxzvf /opt/cni/bin cni-plugins-linux-arm64-v1.1.1.tgz

安装客户端

1)ctr

安装好 containerd 之后,默认就有一个客户端交互工具:ctr。我们来看看这个命令的 Usage:

➜  containerd$ which ctr
/usr/local/bin/ctr
➜  containerd$ ctr help
NAME:
   ctr - 
        __
  _____/ /______
 / ___/ __/ ___/
/ /__/ /_/ /
\___/\__/_/

containerd CLI


USAGE:
   ctr [global options] command [command options] [arguments...]

VERSION:
   v1.6.10

DESCRIPTION:
   
ctr is an unsupported debug and administrative client for interacting
with the containerd daemon. Because it is unsupported, the commands,
options, and operations are not guaranteed to be backward compatible or
stable from release to release of the containerd project.

COMMANDS:
   plugins, plugin            provides information about containerd plugins
   version                    print the client and server versions
   containers, c, container   manage containers
   content                    manage content
   events, event              display containerd events
   images, image, i           manage images
   leases                     manage leases
   namespaces, namespace, ns  manage namespaces
   pprof                      provide golang pprof outputs for containerd
   run                        run a container
   snapshots, snapshot        manage snapshots
   tasks, t, task             manage tasks
   install                    install a new package
   oci                        OCI tools
   shim                       interact with a shim directly
   help, h                    Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --debug                      enable debug output in logs
   --address value, -a value    address for containerd's GRPC server (default: "/run/containerd/containerd.sock") [$CONTAINERD_ADDRESS]
   --timeout value              total timeout for ctr commands (default: 0s)
   --connect-timeout value      timeout for connecting to containerd (default: 0s)
   --namespace value, -n value  namespace to use with commands (default: "default") [$CONTAINERD_NAMESPACE]
   --help, -h                   show help
   --version, -v                print the version
➜  containerd$ 

利用该命令,可以下载镜像,运行容器等。到这里,我们开始能理解,为什么 Docker 并不是运行容器的唯一选项。除了 ctr 之外,还有两款客户端:crictl 和 nerdctl。

2)crictl

如果是配合 kubernetes 使用,笔者推荐安装 crictl。

安装:

VERSION="v1.25.0"

wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-arm64.tar.gz

sudo tar zxvf crictl-$VERSION-linux-arm64.tar.gz -C /usr/local/bin

配置:

cat << EOF | sudo tee /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 2
debug: true
pull-image-on-create: false
EOF

3)nerdctl

如果单独跑容器,则推荐安装 nerdctl。

安装:

wget -c https://github.com/containerd/nerdctl/releases/download/v1.0.0/nerdctl-1.0.0-linux-arm64.tar.gz

sudo tar zxvf nerdctl-1.0.0-linux-arm64.tar.gz -C /usr/local/bin

关于这三款客户端的比较,官方给出的结论如下:


笔者以 crictl 为基准,对比 docker 命令,结果如下:

Function crictl Docker
Attach to a running container crictl attach docker attach
Create a new container crictl create docker create
Run a command in a running container crictl exec docker exec
Display runtime version information crictl version docker version
List images crictl images docker images
Display the status of one or more containers crictl inspect docker inspect
Return the status of one or more images crictl inspecti docker inspect
Return image filesystem info crictl imagefsinfo -
Display the status of one or more pods crictl inspectp -
Fetch the logs of a container crictl logs docker logs
Forward local port to a pod crictl port-forward -
List containers crictl ps docker ps
Pull an image from a registry crictl pull docker pull
Run a new container inside a sandbox crictl run docker run
Run a new pod crictl runp -
Remove one or more containers crictl rm docker rm
Remove one or more images crictl rmi docker rmi
Remove one or more pods crictl rmp -
List pods crictl pods -
Start one or more created containers crictl start docker start
Display information of the container runtime crictl info -
Stop one or more running containers crictl stop docker stop
Stop one or more running pods crictl stopp -
Update one or more running containers crictl update docker update
Get and set crictl client configuration options crictl config -
List container(s) resource usage statistics crictl stats docker stats
List pod resource usage statistics crictl statsp -
Output shell completion code crictl completion -
Checkpoint one or more running containers crictl checkpoint -

crictl 能直接运行 pod,从这个角度来说,它与 Kubernetes 更亲近。

使用样例

这里以 nerdctl 为例,下载镜像并运行容器:

demon@ubuntu2204-master:~$ sudo nerdctl pull docker.io/library/nginx:1.23.1
docker.io/library/nginx:1.23.1: 
index-sha256:2f770d2fe27bc85f68fd7fe6a63900ef7076bc703022fe81b980377fe3d27b70:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:c1c0fedab5e40ba533cbe1f150a49fa3c946ea3fdf4fb4b4cd97ff59930e73d1: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:1db0b6ded6ab40b48b544eafc8d76bc1b06cf25f3774b442de5282c429f4359f:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:8d8f8963a6d4c380ac9d04aa85050c8fbcd5fbe8beb5645a83e002f94ad31a51:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:df8e44b0463f16c791d040e02e9c3ef8ec2a84245d365f088a80a22a455c71e8:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:a5e2bdbd69c78a72e1f1ab7d49c81cfacde6e25b53b4d08a6672b3a6cbcaed58:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:6a8bb8f4c2d138c2f36801334ed80ec1e15a732b45ec57e2c4152703368f03f9:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:37b74cc0b42ab5a4456e1aa6bcbc6965a9fbb03385138cab5c0fc22eeca9eaec:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:71e4fd64785001b8f9dc5448adbc5d79bb854e58a972a0e004e91edd14c1a217:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 22.2s                                                                    total:  52.8 M (2.4 MiB/s)                                
demon@ubuntu2204-master:~$ 
demon@ubuntu2204-master:~$ sudo nerdctl images
REPOSITORY    TAG       IMAGE ID        CREATED           PLATFORM          SIZE         BLOB SIZE
nginx         1.22.1    d317ba7b2ee8    25 minutes ago    linux/arm64/v8    142.5 MiB    52.8 MiB
nginx         1.23.1    2f770d2fe27b    3 minutes ago     linux/arm64/v8    142.5 MiB    52.8 MiB
demon@ubuntu2204-master:~$ 
demon@ubuntu2204-master:~$ sudo nerdctl rmi nginx:1.23.1
Untagged: docker.io/library/nginx:1.23.1@sha256:2f770d2fe27bc85f68fd7fe6a63900ef7076bc703022fe81b980377fe3d27b70
Deleted: sha256:c75eaa0eefd3c60b86bed7b8e234f032b2234382d113a6125c40c553146271fe
Deleted: sha256:d416aac930f73f2f573eefffb35005f779dfc9abc2d63f3083e28012dd1ffefa
Deleted: sha256:caf1f2684a98d5de987369f8046e0066d43eb3d1c9561038c4c3167abce20bdf
Deleted: sha256:7a89740679a77fbc6c143e0ec030cf60c0615985ef8667745cc00cafd9240171
Deleted: sha256:060241690833b97ec43f7f1d9a001160b653e55a0405094c1bea6576315ae7bf
Deleted: sha256:59cd114add79fa5f7f85990eb056de747e200edd7fd410a74ccfaad2c00f2c3e
demon@ubuntu2204-master:~$ 
demon@ubuntu2204-master:~$ sudo nerdctl images
REPOSITORY    TAG       IMAGE ID        CREATED           PLATFORM          SIZE         BLOB SIZE
nginx         1.22.1    d317ba7b2ee8    27 minutes ago    linux/arm64/v8    142.5 MiB    52.8 MiB
demon@ubuntu2204-master:~$ 
demon@ubuntu2204-master:~$ sudo nerdctl run --name nginx-1.22.1 -d --publish 8081:80 nginx:1.22.1
9bf04e8d189fb33c336331a1a17e965a6b6f033d7fa8b4bf1d4e8bedf12348ba
demon@ubuntu2204-master:~$ 
demon@ubuntu2204-master:~$ sudo nerdctl ps 
CONTAINER ID    IMAGE                             COMMAND                   CREATED           STATUS    PORTS                   NAMES
9bf04e8d189f    docker.io/library/nginx:1.22.1    "/docker-entrypoint.…"    14 seconds ago    Up        0.0.0.0:8081->80/tcp    nginx-1.22.1
demon@ubuntu2204-master:~$ 
demon@ubuntu2204-master:~$ sudo nerdctl logs nginx-1.22.1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/11/15 08:30:22 [notice] 1#1: using the "epoll" event method
2022/11/15 08:30:22 [notice] 1#1: nginx/1.22.1
2022/11/15 08:30:22 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2022/11/15 08:30:22 [notice] 1#1: OS: Linux 5.15.0-53-generic
2022/11/15 08:30:22 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1024:1024
2022/11/15 08:30:22 [notice] 1#1: start worker processes
2022/11/15 08:30:22 [notice] 1#1: start worker process 29
2022/11/15 08:30:22 [notice] 1#1: start worker process 30
demon@ubuntu2204-master:~$ 
demon@ubuntu2204-master:~$ curl -i http://localhost:8081
HTTP/1.1 200 OK
Server: nginx/1.22.1
Date: Tue, 15 Nov 2022 08:31:49 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 19 Oct 2022 08:02:20 GMT
Connection: keep-alive
ETag: "634faf0c-267"
Accept-Ranges: bytes
...
...
demon@ubuntu2204-master:~$ 

从整个操作步骤来看,好像与 docker 运行没啥区别,这就是传说中的万变不离其宗吧。

小结

Docker 因为封装应用这个好点子开启了云原生时代,但随着多家厂商的厮杀,Docker 也在不断迭代自身的架构。为了应对危机四伏的局面,Docker 先后搞出了 libcontainer,runc,containerd 等项目,并将其捐献了出去。

就这样,更底层的容器运行时 containerd 诞生了,但这有必要吗?如果不使用容器编排的话,其实 Docker 更友好,操作简单,学习成本低。但容器本身并不值钱,值钱的是容器编排,这才是 PaaS 平台的核心价值。

containerd 偏底层,在大型容器编排领域具备天然的优势,关于这里面的细节,笔者将在 Kubernetes 与 Docker 相爱相杀的故事中再进行剖析,敬请期待。

题图: 来源 kodekloud


  1. 在 LXC 之前,社区就有了容器化隔离技术,比如 OpenVZ 和 Linux-VServer,但它们不是 Linux 官方的技术,也就是说它们需要通过给内核打补丁才能使用,而 LXC 封装系统的理念就源自于它们。 ↩︎