Why Docker and Firewall don’t get along with each other!

In this article, I’m gonna get you through my own experience with interference between Docker and the Firewall on Linux (originally iptables and its interfaces like firewalld or ufw) which took me hours to find out why my firewall rules are not working. If you are a Docker fan too, reading this article may save you a lot of time. Enough with blah blah blah, let’s dive into the story!

The Story

The story began on a beautiful rainy day in which I was deploying a Redis container. Firstly, I ran a container with the official Redis image that was using port 6379, and yes, this port was exposed to the outside world because my application service needed to be able to reach this node by its IP address. The second action was restricting access to this Redis node by defining some simple rules in iptables (my interface was firewalld). I didn’t want someone to just step in and make some trouble for me by attacking this node (flushing it, making it a slave of another Redis, etc). So I simply created a new firewalld rule to block the incoming requests. Guess what? It didn’t work in a very beautiful way!

Challenge accepted!

An angry Docker whale whos attacking the iptables

Struggling!

The docker documentation explains that Docker manipulates firewall rules for network isolation by default. It installs two custom chains called DOCKER and DOCKER-USER . The DOCKER chain contains all necessary docker rules. And the other one,DOCKER-USER chain, is created for user-level rules, and these user-defined rules are applied before any rule that Docker creates automatically.

We aren’t able to add our custom rules directly (to protect Redis) to the INPUT/FORWARD chains as usual. But if we do, they’ll be ignored. Docker in its documentation clearly says so:

Rules added to the FORWARD chain -- either manually, or by another iptables-based firewall -- are evaluated after these chains. This means that if you expose a port through Docker, this port gets exposed no matter what rules your firewall has configured.

So I tried to add my Redis restriction rule to this DOCKER-USER chain:

# I droped incoming traffics from all IP addresses except an authorized one (1.2.3.4 for example)
iptables -I DOCKER-USER -i eth0 --dport 6379 -s 1.2.3.4 -j ACCEPT
iptables -I DOCKER-USER -i eth0 --dport 6379 -j DROP

Unfortunately, it didn’t work too! I tried to remove and add it again multiple times but there was no difference with the final result. So I took another shot with Docker's official example:

iptables -I DOCKER-USER -i eth0 ! -s 1.2.3.4 -j DROP

Yohu, it worked!! But there were still some limitations:

• The restriction was applied only to the IP address, not the IP and Port.
• I couldn’t add more trusted IPs. (technically, it is possible with ipset , but it brings more complexity with itself which wasn’t my option— but if you want to do it anyway, check this link).
• I had to put my rules directly to the iptables but I still wanted to use an easy-to-use interface (firewalld).

Because of the mentioned reasons, I decided to find a better solution that not only works with iptables, but also works with its interfaces.

Final Solution

I was sick of searching for hours and finding nothing. Yes you’re right, the problem was the Docker and iptables not firewalld and that’s an answer to “why I couldn’t find a solution”.

I came back to the same Docker and Iptables documentation page and at the end of this page, I saw a section with the title “Prevent Docker from manipulating iptables”. It was the game-changer of this story! Let’s break down what it really means.

The Docker gives you the option to disable iptables rules manipulation with a flag; the iptables=false. But it also mentions that you may face some networking issues if you use it. Don’t worry, I’ve covered that in this article.

Anyway, it seemed a legit solution to me. So I created a file in the following path:

vim /etc/systemd/system/docker.service.d/noiptables.conf

And put the instructions to disable iptables manipulation at the service level. By making changes to the docker service, it won’t get back to the default settings after every restart:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=false

It was time to reload the system daemon (because of service config modification) and restart the Docker service:

sudo systemctl daemon-reload
sudo systemctl restart docker

That was all! After these steps, I could configure my firewall rules!

Some notes

After disabling iptablesmanipulation, make sure that:

• Your containers have access to the internet. A simple ping from inside the container would be good.
• Your containers can communicate with each other on the same network.

If you’re experiencing one of the above issues, check this, this, and this links.

Sören Metje has also covered more detail in his article “How to Secure a Docker Host Using Firewalld”. He has also mentioned solution for ufw users.

If this article helped you in any way, please give it a clap.

Happy DevOps! 🎉