Diagnosing SSH Connectivity: Firewall Conflicts and Hidden Bans

A founder's multi-stage debugging journey reveals how conflicting firewall rules, systemd service interactions, and automated security tools can silently block SSH access, even when services appear online.

An attempt to SSH into a server failed, despite all other services, including Gitea, Nextcloud, and a k3s cluster, remaining operational. The founder's initial nmap -p 22 2.27.42.100 scan returned PORT STATE SERVICE 22/tcp filtered ssh, indicating packets reached the server but were silently dropped, pointing to a firewall issue. This initial state set the stage for a complex, multi-layered debugging process.

Initial Diagnosis: nmap and iptables

The first step involved confirming the filtered status via nmap. This output is crucial: filtered means a firewall is actively dropping packets, as opposed to closed (no service listening) or refused (connection actively rejected). The founder then checked the INPUT chain using iptables, observing an ACCEPT tcp dpt:22 rule at position 2. This rule initially appeared correct, suggesting SSH traffic should be permitted. However, the presence of Chain INPUT (policy DROP) meant that if no other rule matched, traffic would be discarded. The ACCEPT rule's placement and apparent correctness proved misleading.

Uncovering Firewall Conflict with nftables

The iptables output was not the full picture. A deeper inspection using nft list ruleset | grep -E "22|drop|DROP" revealed a type filter hook input priority filter; policy drop; setting. Critically, a tcp dport 22 counter packets 0 bytes 0 accept rule was present, but showed zero packets, confirming connections were not even reaching it. The core issue emerged: the server ran both iptables-nft (managed by kube-router and UFW) and native nftables simultaneously. These systems conflicted, with kube-router constantly reconciling and overwriting firewall states. A warning in the configuration, # Warning: table ip filter is managed by iptables-nft, do not touch!, had been present but overlooked. The deployment of a Gitea runner days prior had triggered a reconciliation that corrupted the rule ordering.

First Fix Attempt: Native nftables Priority Override

To bypass the conflict, the founder implemented a native nftables solution with a high priority. A new inet table named ssh_rescue was created, and a chain within it was set with type filter hook input priority -150. This priority ensures the rule executes before any standard filter chains (priority 0) and before kube-router's management, which only affects the iptables-nft layer. The command nft add rule inet ssh_rescue input tcp dport 22 accept was then used to explicitly accept SSH traffic. This immediately restored SSH connectivity.

Second Issue: systemd and fail2ban Interference

The initial fix was short-lived. After making the nftables configuration persistent via a systemd service, SSH died again. The systemctl start nftables command had loaded the saved /etc/nftables.conf, which still contained kube-router's warning comments and corrupted the nftables state, undoing the manual fix. After re-adding the table manually, the founder discovered fail2ban had installed itself as a dependency. fail2ban had banned two IPs due to bots hammering port 22, and the founder's own repeated failed attempts were caught in this. Running fail2ban-client unban --all restored connectivity. However, SSH died again shortly after, even with fail2ban disabled.

The Unresolved Root Cause: VPN Conflict

During further debugging, the founder realized a critical detail: their self-hosted VPN, located on a server in Finland, was relevant. The post indicates that the founder had been "compl"—presumably implying a change or issue with the VPN setup or connection originating from a different location. The signal payload ends here, leaving the specific resolution of this VPN-related root cause undescribed. The prior debugging steps, while valid for their specific issues, were ultimately addressing symptoms rather than the underlying problem.

WHAT WE'D CHANGE

The founder's debugging journey highlights common pitfalls in server management, but a comprehensive playbook requires additional steps and considerations. The most significant gap is the incomplete resolution of the "actual reason"—the VPN conflict. A full diagnostic process would necessitate detailing how the VPN configuration was identified as the root cause and the specific steps taken to resolve it. Without this, the playbook remains incomplete regarding the ultimate issue.

For firewall conflicts, the founder's method of using nftables with a high priority is a viable workaround for immediate access. However, a more robust solution would involve consolidating firewall management to a single system or explicitly configuring kube-router to permit SSH traffic without conflict. Proactive monitoring for iptables/nftables rule changes, especially in containerized environments like k3s, could prevent such issues from escalating. Tools that diff firewall rulesets over time could flag unauthorized or conflicting changes.

Regarding fail2ban, its discovery late in the process indicates a need for earlier consideration in a debugging checklist. When SSH issues arise, fail2ban logs should be among the first checks, particularly after multiple failed login attempts. Implementing a whitelist for known administrative IPs in fail2ban configurations can prevent self-inflicted bans during debugging. Furthermore, the systemd service issue underscores the importance of thoroughly testing configuration persistence. Any manual fix should immediately be followed by a test of its persistence across service restarts and system reboots, verifying that systemd units load the intended configuration without corruption or unintended overrides.

LANDING

Server-side debugging often presents as a layered problem, where initial symptoms mask deeper, interconnected issues. This founder's experience demonstrates that even with seemingly correct firewall rules, conflicts between network management tools, automated security systems, and external network configurations can create persistent connectivity failures. A systematic approach, moving from network diagnostics to firewall rule inspection, then to service-level checks, and finally to external network factors, is essential for isolating and resolving such complex problems. The process demands meticulous verification at each stage, as a partial fix can obscure the true root cause. This incident underscores the necessity of understanding the full network stack, from the kernel to external VPNs. The founder's experience highlights that the most effective debugging often involves stepping back to consider all potential layers, even those initially dismissed.

Pull quote: “”