Why Docker and Firewall don’t get along with each other!

In this article, I’m gonna get you through my own experience about interference between the Docker and the Firewall on Linux (originally iptables and its interfaces like firewalld or ufw) which took me hours to find out why my firewall rules are not working. If you are a Docker fan too, reading this article may save you a lot of time. Enough with blah blah blah, let’s dive into the story!

The Story

The story began on a beautiful rainy day in which I was deploying a Redis container. Firstly, I ran a container with the official Redis image that was using port 6379, and yes, this port was exposed to the outside world because my application service needed to be able to reach this node by its IP address. The second action was restricting the access to this Redis node by defining some simple rules in iptables (my interface was firewalld). I didn’t want to someone just step in and make some troubles for me by attacking this node (flushing it, making it slave of another Redis or, etc). So I simply created a new firewalld rule to block the incoming requests. Guess what? It didn’t work in a very beautiful way!

Challenge accepted… !

An angry Docker whale whos attacking the iptables

Struggling!

The docker documentation explains that Docker manipulates firewal rules for network isolation by default. It installs two custom chains called DOCKER and DOCKER-USER . The DOCKER chain contains all necessary docker rules. And the other one,DOCKER-USER chain, is created for user-level rules, and these user-defined rules are applied before any rule that Docker creates automatically.

We aren’t able to add our custom rules directly (to protect Redis) to the INPUT/FORWARD chains as usual. But if we do, they’ll be ignored. Docker in its documentation clearly says so:

Rules added to the FORWARD chain -- either manually, or by another iptables-based firewall -- are evaluated after these chains. This means that if you expose a port through Docker, this port gets exposed no matter what rules your firewall has configured.

So I tried to add my Redis restriction rule to this DOCKER-USER chain:

# I droped incoming traffics from all IP addresses except an authorized one (1.2.3.4 for example)
iptables -I DOCKER-USER -i eth0 --dport 6379 -s 1.2.3.4 -j ACCEPT
iptables -I DOCKER-USER -i eth0 --dport 6379 -j DROP

Unfortunately, it didn’t work too! I tried to remove and add it again multiple times but there was no difference with the final result. So I took another shot with Docker official example:

iptables -I DOCKER-USER -i eth0 ! -s 1.2.3.4 -j DROP

Yohu, it worked!! But there were still some limitations:

  • The restriction was applied only on the IP address, not the IP and Port.
  • I couldn’t add more trusted IPs. (technically, it is possible with ipset , but it brings more complexity with itself wich wasn’t my option— but if you want to do it anyway, check this link).
  • I had to put my rules directly to the iptables but I still wanted to use an easy to use interface (firewalld).

Because of the mentioned reasons, I decided to find a better solution that not only works with iptables, but also works with its interfaces.

Final Solution

I was sick of searching for hours and finding nothing. Yes you’re right, the problem was the Docker and iptables not firewalld and that’s an answer to “why I couldn’t find a solution”.

I came back to the same Docker and Iptables documentation page and at the end of this page, I saw a section with the title “Prevent Docker from manipulating iptables”. It was the game-changer of this story! Let’s break it down what it really means.

The Docker gives you an option to disable iptables rules manipulation with a flag; the iptables=false. But it also mentions that you may face with some networking issues if you use it. Don’t worry, I’ve covered that in this article.

Anyway, it seemed a legit solution to me. So I created a file in the following path:

vim /etc/systemd/system/docker.service.d/noiptables.conf

And put the instructions to disable iptables manipulation at the service level. By making changes to the docker service, it won’t get back to the default settings after every restart:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --iptables=false

It was time to reloading system daemon (because of service config modification) and restarting the Docker service:

sudo systemctl daemon-reload
sudo systemctl restart docker

That was all! After these steps, I could configure my firewall rules!

Some notes

After disabling iptablesmanipulation, make sure that:

  • Your containers have access to the internet. A simple ping from inside the container would be good.
  • Your containers can communicate with each other on the same network.

If you’re experiencing one of the above issues, check this, this and this links.

If this article helped you in any way, please give it a clap.

Happy DevOps! 🎉

Back-End Engineer and DevOps enthusiast. Startup Lover.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store