Working with Docker has become a staple for developers across different platforms, especially when it comes to packaging applications into containers. One thing that’s often overlooked, though, is how much control you can have over Docker’s image-pushing process. If you’ve ever been frustrated by slow uploads or wanted to optimize how Docker pushes layers to your container registry, you’re not alone. Docker does this operation behind the scenes, but with a little tweak, you can enhance the performance or fine-tune the process to better suit your network or environment.
Thank me by sharing on Twitter 🙏
In this post, I’m going to walk through an approach that I found quite useful: controlling the number of concurrent uploads when pushing Docker images. By limiting the number of concurrent uploads, we can prevent Docker from overwhelming a network or Docker registry, which can be a game-changer, especially when you’re working with large images or have bandwidth restrictions. While Docker defaults to pushing layers in parallel, we can fine-tune this behavior to fit our needs by adding a configuration in daemon.json on Linux or through Docker Desktop settings on macOS.
Why Control Docker Pushes?
Docker pushes its image layers in parallel to speed up the process, but sometimes this is counterproductive. If you’re on a slower or constrained network, or if your Docker registry has strict upload limits, the default behavior might flood the connection, leading to throttling or even failed uploads. This can particularly be the case with cloud services or environments that enforce API rate limits.
By controlling how Docker pushes image layers, we can:
- Prevent overwhelming the network
- Reduce registry-side throttling issues
- Gain more predictable upload speeds
- Avoid failed uploads due to concurrency
Setting this up is straightforward. It’s as simple as updating a configuration file or modifying Docker Desktop settings, depending on your operating system. Let’s explore how to do this on both Linux and macOS.
BENGOO G9000 Stereo Gaming Headset for PS4 PC Xbox One PS5 Controller, Noise Cancelling Over Ear Headphones with Mic, LED Light, Bass Surround, Soft Memory Earmuffs (Blue)
$25.99 (as of December 21, 2024 08:38 GMT +00:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)
HP 67XL Black High-yield Ink Cartridge | Works with HP DeskJet 1255, 2700, 4100 Series, HP ENVY 6000, 6400 Series | Eligible for Instant Ink | One Size | 3YM57AN
$29.89 (as of December 21, 2024 08:38 GMT +00:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)
Co-Intelligence: Living and Working with AI
$17.79 (as of December 21, 2024 19:39 GMT +00:00 - More infoProduct prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on [relevant Amazon Site(s), as applicable] at the time of purchase will apply to the purchase of this product.)Configuring Docker Pushes on Linux
On Linux-based systems like Ubuntu, controlling how Docker pushes image layers requires editing the Docker daemon configuration file. This file tells Docker how to behave, and luckily, it’s very easy to modify.
Step 1: Open the Docker Configuration File
First, you’ll want to locate the Docker daemon configuration file. It’s typically found in /etc/docker/daemon.json. If this file doesn’t exist, don’t worry — you can create it.
To open or create the file, use a text editor like nano:
sudo nano /etc/docker/daemon.jsonStep 2: Add the Concurrent Upload Setting
In this file, you’ll add a simple setting to control the number of concurrent uploads. This setting is named "max-concurrent-uploads", and we’ll set it to 1 to restrict Docker to uploading one layer at a time.
Here’s what the configuration might look like:
{
"max-concurrent-uploads": 1
}If you already have some settings in this file (such as enabling debugging or other options), simply add this new setting to the JSON structure, making sure commas are correctly placed between options.
For example, if you also have debugging enabled, your file might look like this:
{
"debug": true,
"max-concurrent-uploads": 1
}Step 3: Restart the Docker Service
After saving your changes, you’ll need to restart the Docker service for the changes to take effect. You can do this using the following command:
sudo service docker restartOnce the service restarts, Docker will only upload one layer at a time when pushing an image to a registry.
Configuring Docker Pushes on macOS (Docker Desktop)
If you’re using macOS, the steps are slightly different since Docker runs inside Docker Desktop, and you’ll need to access the settings through the GUI. Thankfully, Docker Desktop makes this process just as simple.
Step 1: Open Docker Desktop Settings
First, make sure Docker Desktop is running. Then, click on the Docker icon in your menu bar (top-right corner of your screen) and select Settings (or Preferences on older versions of Docker Desktop).
Step 2: Modify the Docker Engine Configuration
Once inside the settings, navigate to the Docker Engine tab. This section allows you to edit Docker’s daemon.json configuration directly, similar to what we did on Linux.
In the text editor that appears, add the following line to the existing JSON structure:
{
"max-concurrent-uploads": 1
}
As with Linux, if there are other settings in place, simply add this new line without removing the existing configurations. For example:
{
"debug": true,
"max-concurrent-uploads": 1
}Step 3: Apply and Restart Docker
Once you’ve made your changes, click the Apply & Restart button at the bottom of the screen. Docker Desktop will restart, and from that point on, Docker will limit itself to one concurrent layer upload at a time.
When to Use This Setting
You might be wondering when it’s appropriate to limit concurrent uploads, especially if Docker defaults to parallel uploads for better speed. Here are a few scenarios where controlling the number of uploads might be beneficial:
- Low bandwidth environments: If you’re pushing images over a network with limited bandwidth, restricting the number of uploads can help prevent your connection from being overwhelmed, resulting in faster and more reliable pushes.
- Registry API rate limiting: Some Docker registries (especially hosted ones like AWS ECR or Docker Hub) might enforce API rate limits, meaning that sending too many requests (in this case, too many parallel uploads) could result in errors or throttling. Limiting concurrent uploads can prevent this from happening.
- Predictable resource usage: If you want to make sure your push operation uses a predictable amount of bandwidth or system resources, limiting the number of concurrent uploads can help stabilize network usage and avoid spikes.
Conclusion
Configuring Docker to control the number of concurrent uploads is a simple but powerful way to optimize your container pushes. Whether you’re dealing with a slow network, API limits, or simply want more predictable performance, this tweak can improve the reliability and speed of your Docker image pushes. On Linux, it’s as simple as adding a line to the daemon.json file, and on macOS, Docker Desktop provides an intuitive way to make the same adjustment.
By limiting Docker to one upload at a time, you can prevent unnecessary failures, avoid network congestion, and enjoy smoother interactions with your container registries. This setting might not be needed in every environment, but when it is, it can make a world of difference.
