To benchmark the performance of container runtimes like containerd and dockerd, you can evaluate key metrics around startup speed, resource efficiency, and scalability. This guide walks through the essential benchmarks, tools, and step-by-step instructions needed to get accurate and actionable data.

Tools and Setup for Benchmarking:

System Requirements

• Use a machine (bare metal or virtual) with sufficient CPU, RAM, and disk space to handle container loads.
• Install both containerd and dockerd on the same system for consistent, fair comparisons.

Command-line Tools

• time: Measures execution duration for various tasks.
• docker and ctr commands: Manages containers within dockerd and containerd, respectively.
• htop, top, or ps: Monitors CPU and memory usage.
• hey: Executes HTTP load tests for containerized applications.

Benchmark Categories

1. Startup Time
2. Resource Utilization (Memory and CPU)
3. Image Pull Performance
4. Container Lifecycle Operations
5. Networking Performance
6. Disk I/O Performance

Let’s go into each benchmark, including sample commands and expected results.

1. Container Startup Time
Measure the time taken for containers to start with both runtimes.

• containerd: Use `ctr` for container management.

for i in {1..10}; do                                             ✔ 
    { time sudo ctr run -d  docker.io/library/nginx:latest nginx_$i; } 2>> containerd_times.txt 
done

Result:

• dockerd: Use `docker` to run the container.

for i in {1..10}; do { time docker run -d nginx:latest; } 2>> dockerd_times.txt done

Result:

Output Comparison:

Conclusion:

Containerd is generally faster than Docker, with a runtime of 0.07 seconds compared to Docker’s 0.13 seconds, thanks to its lightweight architecture.

2. Resource Utilization (Memory and CPU)
Use `htop`, `top`, or `ps` to measure the CPU and memory usage of containers managed by containerd and dockerd.

For our example we gonna use ps command

• containerd:

ps -C containerd -o rss= | awk '{sum += $1} END {print sum/1024 " MB"}'bas

- dockerd:

ps -C dockerd -o rss= | awk '{sum += $1} END {print sum/1024 " MB"}'

Output:

Conclusion:

• containerd is likely to use less memory and CPU due to its streamlined architecture compared to dockerd.

3.Image Pull Performance
Measure the time taken to pull a container image for both runtimes.

• containerd:

time sudo ctr image pull docker.io/library/nginx:latest

Result:

• dockerd:

time docker pull nginx:latest

Result:

Conclution:
- Both runtimes use the same container image format, so performance should be bit similar. However, you might observe slight differences showing better perf for containerd.

4.Container Lifecycle Operations
Measure how long it takes to start, stop, and remove containers.

• Start a container:

— Containerd:

time ctr run -d docker.io/library/nginx:latest test-container

Result:

— Dockerd:

time docker run -d nginx:latest

• Stop a running container:

— Containerd:

time sudo ctr tasks kill test-container

Result:

— Dockerd:

docker stop <container-id>

Output:
- Measure the time for each operation (start, stop, and remove). containerd will likely show quicker response times for basic operations due to its lightweight nature.

Key Comparison Points:

Conclusion:

• Containerd is faster for container start operations with less total execution time (0.139s vs 0.534s).
• Docker’s docker stop operation is slower because stopping a container involves more internal management, but the task is simpler in terms of CPU usage.

5. Networking Performance
Run a network benchmark to test how efficiently both runtimes handle network traffic.

• Set up an Nginx container with both runtimes:
   — containerd:

sudo ctr run --rm --net-host docker.io/library/nginx:latest test-container

— dockerd:

docker run -rm -p 8080:80 nginx:latest

• Use `hey` to send HTTP requests to the container:
   — Containerd

hey -n 10000 -c 400 http://localhost:80

Output:

— Dockerd

hey -n 10000 -c 400 http://localhost:8080

Output:

Conclusion:

• Performance: Containerd outperformed Docker in every metric tested. It achieved higher RPS, lower average latency, and significantly lower slowest latency. Containerd’s response times were more consistent, especially at the 90th and 99th percentiles.
• Scalability: Containerd demonstrated better scalability, handling twice as many requests per second with lower latency compared to Docker.

Containerd provides better networking performance, handling requests more efficiently with lower overhead. Docker showed more significant latency for slower requests, impacting its overall performance when handling high volumes of traffic.

— -

6. Disk I/O Performance
Benchmark disk I/O performance by writing and reading files from the containers.

• containerd:

sudo ctr run --rm --mount type=bind,src=/tmp,dst=/mnt,options=rbind:rw docker.io/library/alpine:latest test-container dd if=/dev/zero of=/mnt/test bs=1M count=100

Output:

• dockerd:

docker run --rm -v /tmp:/mnt alpine dd if=/dev/zero of=/mnt/test bs=1M count=100

Output:

Conclusion:

containerd: Total time: 6.14 seconds , Throughput: 814.1 MB/s

dockerd: Total time: 21.65 seconds, Throughput: 230.9 MB/s

- containerd typically performs better due to lower disk I/O overhead, as it focuses on managing containers efficiently without the added layers that dockerd brings.

These benchmarks provide a clear comparison of containerd and dockerd across critical operational metrics, helping to identify the runtime best suited for high-performance environments:

• Startup Time: containerd demonstrates faster container startup times, highlighting its efficiency in environments requiring rapid container deployment.
• Resource Utilization: containerd consistently uses memory and CPU more efficiently than dockerd, thanks to its streamlined, lightweight architecture, which can be beneficial in resource-constrained or multi-tenant environments.
• Lifecycle Operations: With lower latency for basic operations (start, stop, and remove), containerd offers faster lifecycle management, ideal for workloads requiring frequent container state changes.
• Networking and Disk I/O Performance: containerd shows a slight edge in both network and disk I/O benchmarks. Its reduced operational overhead translates to better handling of network traffic and higher disk I/O throughput, making it more adaptable for data-intensive applications.

This comprehensive data highlights that containerd generally outperforms dockerd across multiple dimensions of performance and scalability. For cloud-native and high-demand environments, containerd stands out as the optimal choice due to its speed, efficiency, and lower overhead.