SSH Security with FIDO2: Hardening Linux Servers Using Hardware Keys

I remember the exact moment I realized my .ssh folder was a liability. I had a dozen private keys scattered across laptops, jump boxes, and a dusty backup drive, each protected by nothing more than a passphrase I’d definitely reused somewhere else. If you're serious about Linux server hardening, you know that relying on software-based RSA or Ed25519 keys is a gamble—if your machine is compromised, your keys are effectively exfiltrated.

The solution is hardware-backed authentication. By leveraging FIDO2/U2F hardware security keys with OpenSSH, you move the private key material off your machine entirely. It’s a massive upgrade for your security posture.

Why move to FIDO2/U2F for SSH security?

Traditional SSH keys are "portable." If a malicious actor gets hold of your private key file, they have the keys to the kingdom. With FIDO2-backed keys, the private key is generated inside the hardware (like a YubiKey or Google Titan). The private portion never leaves the device. Even if someone steals your laptop, they cannot duplicate your SSH key. They’d need the physical token and, usually, the PIN you set on that token.

We’ve previously discussed Linux server hardening: Automate audits with Lynis and fail2ban to catch misconfigurations, but authentication is your first line of defense. If your front door is weak, no amount of auditing will save you.

Getting started with hardware security keys

First, ensure your environment supports it. You need OpenSSH 8.2 or higher on both your local machine and the remote server. You can check your version with ssh -V.

1. Generate the FIDO2 key pair

On your local machine, plug in your hardware security key and run the following command:

Bash
ssh-keygen -t ed25519-sk -O resident -O verify-required -f ~/.ssh/id_ed25519_sk

The -sk (security key) suffix is the magic here. The -O resident flag tells the key to store the handle on the hardware, which is incredibly convenient if you move between machines. The -O verify-required flag forces the device to verify the user presence (like a touch or a PIN).

2. Configure the remote server

Copy your public key to the remote host as you normally would:

Bash
ssh-copy-id -i ~/.ssh/id_ed25519_sk.pub user@your-server-ip

Once the key is in ~/.ssh/authorized_keys, you need to ensure the server is ready to accept it. In modern Linux distributions (Ubuntu 22.04+, RHEL 9+, Debian 11+), this is enabled by default. If you run into issues, check your sshd_config:

Bash
# Ensure these are present
PubkeyAuthentication yes

The trade-offs and lessons learned

When I first rolled this out, I made a mistake: I didn't test my recovery path. I generated a resident key on a new YubiKey and wiped my old software keys before confirming I could SSH into my emergency console.

Pro-tip: Always keep a secondary, non-FIDO2 key as a fallback in your authorized_keys file, and keep that key in a physical safe. If you lose your FIDO2 key, you will be locked out of your own infrastructure.

Also, don't forget that hardware keys can be sensitive to environmental factors. I once had a cheap USB-C hub that caused my YubiKey to disconnect mid-handshake. It took me about 45 minutes to debug that the issue wasn't the protocol, but a buggy hardware interface.

Advanced SSH Security: Beyond the Basics

If you want to take this further, combine this with other layers of security. I’ve found that using Linux Security: File Integrity Monitoring with AIDE and Systemd Timers acts as a great "catch-all" if someone ever manages to bypass your SSH authentication through a zero-day exploit.

When you're dealing with larger fleets, you might eventually look into SSH Certificates or managing keys via Kubernetes Secret Management with HashiCorp Vault and ESO Guide, but for most solo engineers or small teams, FIDO2 is the "gold standard" for individual access.

Frequently Asked Questions

Q: Do I need to re-generate keys for every server? A: No. Because you used the resident flag, you can plug your key into a new machine and run ssh-add -K. It will pull the key handle from the hardware, and you're ready to go.

Q: What if I lose my hardware security key? A: You’re locked out. This is why you must have a "break-glass" recovery key stored in a physical safe. Never rely on a single hardware token for production access.

Q: Does this work with older Linux distributions? A: If you are running something ancient (like CentOS 7), you’ll likely need to compile a newer version of OpenSSH from source. I wouldn't recommend it—if you're on a legacy system, upgrade the OS instead of hacking the SSH daemon.

I'm still tinkering with how to handle FIDO2 keys in automated CI/CD pipelines. Right now, I still use standard service accounts for automation, which feels less secure than my personal access. There’s always another layer to peel back.