In this tutorial, I’ll explain how to protect your public-facing Linux server and Nginx web server from common threats, including brute-force and DoS attacks.

Servers exposed to the internet often face these types of attacks, which can disrupt service and compromise security.

These attacks can seriously disrupt a server’s performance and security. By implementing defense in depth, protection mechanism like Fail2ban, Nginx limit_req, and iptables are so important.

What is Brute-Force Attack and Denial-of-Service Attack?

Brute-Force Attack

This is when an attacker tries to guess a user’s password or other sensitive information by trying many different combinations very quickly. It’s like repeatedly trying different keys to unlock a door until one works. If successful, they can access private areas of your server.

Denial-of-Service (DoS) Attack

In a DoS attack, the attacker tries to overwhelm a server by flooding it with requests. A DoS attack can make your server slow or even crash, preventing real users from accessing it

Fail2ban

Fail2Ban is an intrusion prevention software framework. Written in the Python programming language, it is designed to prevent brute-force attacks.

How to Install fail2ban on Debian / Ubuntu

sudo apt install fail2ban

How to Setup fail2ban

After installing fail2ban, the next steps involve configuring it to protect your server effectively. Here’s how to set it up and customize it for maximum security:

1. Configure the Default Jail Settings

The main configuration file is located at /etc/fail2ban/jail.conf, but it’s better practice to override settings in /etc/fail2ban/jail.local to avoid overwriting when updating Fail2ban. Create or edit the jail.local file:

sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
sudo nano /etc/fail2ban/jail.local

Common parameters to configure:

  • bantime: Duration in seconds an IP is banned. Default value: 600 (600 seconds or 10 minutes)
  • findtime: Time window for considering repeated failed attempts. Default value: 600 (600 seconds or 10 minutes)
  • maxretry: Number of failed attempts before a ban. Default value: 5

I personally prefer to use these default values.

2. Enable and Customize Jails for SSH Protection

In the jail.local file, find the [sshd] section to enable SSH protection. By default, SSH protection is enabled for Debian and Ubuntu, in this case I just want to make it explicit with enabled = true.

[sshd]
enabled = true

# The remaining configurations, I prefer to use default values
...

Adjust the parameters as needed for your specific security policy.

3. Enable nginx-limit-req and nginx-botsearch Jails

Fail2ban can monitor other services beyond SSH. Some additional common jails:

  • [nginx-limit-req]: This jail works by identifying IPs that exceed a request threshold within a given time window, effectively catching high-frequency requests typical of DoS attacks. Fail2ban will then ban these IPs temporarily, minimizing the impact of the flood.
  • [nginx-botsearch]: This jail helps by identifying common malicious patterns, such as bots attempting to access sensitive or admin paths repeatedly. Fail2ban will block these IPs automatically, reducing the need for manual intervention with iptables.

To enable these, make sure they’re set to enabled = true in the jail.local file.

[nginx-limit-req]
enabled = true

# The remaining configurations, I prefer to use default values
...

[nginx-botsearch]
enabled = true

# The remaining configurations, I prefer to use default values
...

4. Restart Fail2ban and Verify Configuration:

After editing the configuration, restart Fail2ban to apply the changes:

sudo systemctl restart fail2ban

Check the status to confirm that it’s running and protecting the desired services:

sudo fail2ban-client status

You should see a list of enabled jails, each monitoring the specified service logs.

5. Test Fail2ban Functionality:

Simulate a failed login attempt to see if Fail2ban blocks the IP after repeated attempts (you can try logging in with an incorrect SSH password multiple times). After reaching the maxretry limit, Fail2ban should ban the IP.

6. View Active Bans and Unban IPs if Necessary:

To see currently banned IPs:

sudo fail2ban-client status sshd

To unban an IP, use:

sudo fail2ban-client set sshd unbanip <IP_ADDRESS>

Fail2ban will now be actively monitoring and banning IPs based on the rules you configured, helping secure your server against brute-force and other common unauthorized access attempts.

Nginx limit_req

The Nginx ngx_http_limit_req_module module (0.7.21) is used to limit the request processing rate per a defined key, in particular, the processing rate of requests coming from a single IP address. The limitation is done using the “leaky bucket” method.

http {
    limit_req_zone $binary_remote_addr zone=one:10m rate=10r/s;

    ...

    server {

        ...

        location / {
            # delay up to 2s before processing
            limit_req zone=one burst=5 delay=2;
        }

Nginx limit_req Nodelay

The nodelay setting in Nginx’s limit_req directive can significantly affect how requests are throttled. Here’s a breakdown to help decide whether or not to use nodelay:

What nodelay Does

  • Without nodelay (the default setting), Nginx will queue excess requests and serve them at the rate specified in limit_req_zone. This works well for spreading out requests rather than dropping them immediately.
  • With nodelay, Nginx immediately rejects requests that exceed the limit, without queuing. This makes it more strict, immediately responding with a 503 (Service Unavailable) error if the limit is reached.

When to Use nodelay

Consider enabling nodelay if:

  • You’re dealing with high traffic or frequent burst attacks and want immediate rejection to conserve server resources.
  • You have endpoints where latency is critical, and you prefer dropping excess requests rather than queuing them (e.g., API endpoints with strict rate limits).

When Not to Use nodelay

Keep nodelay disabled (default) if:

  • You want to give legitimate users a chance to access the site, even under load.
  • Your server can handle the request queue, and you’d rather not drop requests unless absolutely necessary (e.g., user-facing sites where some waiting is acceptable).

Example Configuration

If you decide to use nodelay, here’s how to apply it:

limit_req_zone $binary_remote_addr zone=one:10m rate=5r/s;

server {
    location /api/ {
        limit_req zone=one burst=10 nodelay;
    }
}

Simulate Flood to Web Server

You can simulate a request flood by using ab or ApacheBench:

# Install ab
sudo apt install apache2-utils

# Run ab
# -n: Total number of requests to perform.
# -c: Number of concurrent requests to keep open at a time
ab -n 1000 -c 100 https://example.com/

This command sends 1k requests with 100 requests concurrently.

iptables

Using iptables, you can add an extra layer of protection to your server by controlling incoming and outgoing traffic. Here’s a straightforward setup to help secure an Nginx server against common attacks.

1. Install iptables (if not already installed)

Most Ubuntu systems come with iptables pre-installed. Check if it’s installed with:

iptables -v

If not installed, install it with:

sudo apt install iptables

2. Allow SSH, HTTP, and HTTPS Traffic

Allow SSH, HTTP, and HTTPS traffic to ensure you can manage the server and serve web content.

# Allow SSH (port 22) for remote access
sudo iptables -A INPUT -p tcp --dport 22 -j ACCEPT

# Allow HTTP (port 80) and HTTPS (port 443) for web traffic
sudo iptables -A INPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 443 -j ACCEPT

3. Allow Rules for APT

# Allow Outbound HTTP and HTTPS Traffic
# apt uses HTTP/HTTPS to fetch packages
sudo iptables -A OUTPUT -p tcp --dport 80 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 443 -j ACCEPT

# Allow DNS Resolution
# apt needs DNS to resolve domain names to IP addresses
sudo iptables -A OUTPUT -p udp --dport 53 -j ACCEPT
sudo iptables -A OUTPUT -p tcp --dport 53 -j ACCEPT

# Ensure related and established connections are allowed
sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

4. Limit SSH Connections

If you want to limit SSH connection attempts to prevent brute-force attacks, set a rate limit:

sudo iptables -A INPUT -p tcp --dport 22 -m limit --limit 3/min -j ACCEPT

This rule allows a maximum of 3 SSH connection attempts per minute. Excess attempts will be blocked.

5. Block All Other Traffic (Be Careful)

Block all other incoming traffic for security, except those you specifically allowed. Be careful with this rule, as it blocks all incoming connections not specifically allowed above.

sudo iptables -A INPUT -j DROP

6. Check and Save Your Rules

View current rules with:

sudo iptables -L

Save the rules to apply, so the rules will not lost after rebooting. Use iptables-persistent:

sudo apt install iptables-persistent
sudo netfilter-persistent save

7. How to Delete a Rule

# Identify the Rule’s Line Number
sudo iptables -L INPUT --line-numbers

# Delete the Rule by Its Line Number
sudo iptables -D INPUT <line-number>

Last, But Not Least

Those are all the steps, but remember, implementing Fail2ban, Nginx limit_req, and iptables won’t make your server 100% secure. Keep monitoring your server and web application regularly.