When your server is encrypted, you have to type a password each time it boots - in order to do that, you need a monitor and a keyboard hooked up to the server. You can imagine that it would be super annoying to go to the server physical location just to type a password. In the case of a server being off-site, encryption would be a deal breaker. But there is a way around it - SSH.

Booting process

When PC is powered on, it first runs BIOS/EFI that performs hardware initialization, then it tries to find operating system to load. In the case of Linux system, bootloader will load a Linux kernel and an image called “initial RAM disk” or initrd - it’s an image containing essential programs and tools to mount disks, decrypt them and start the PID1 process (these days, systemd is most distros’ choice). Any interaction with the user goes through a terminal - so typically a monitor and a keyboard are needed to talk to the PC.

While this is normal behaviour, in the case of servers, you typically does not have either a keyboard or a monitor attached. When the system is not encrypted, this is fine - boot will proceed to the point where SSH daemon starts and you can connect to the machine remotely. But when system is encrypted, problem appears - how you enter a password remotely? By default Fedora does not provide means for spawning SSH daemon by initrd, but there’s a project called dracut-sshd that works with dracut (tool that generates initrd after Linux kernel updates) to include SSH server inside initrd.

initrd & dracut

Following dracut’s own description:

dracut is an event driven initramfs infrastructure. dracut (the tool) is used to create an initramfs image by copying tools and files from an installed system and combining it with the dracut framework, usually found in /usr/lib/dracut/modules.d.

Since it was developed by Red Hat folks, it’s also used in Red Hat’s distro family (Fedora, CentOS, RHEL). By design, dracut defines so called “modules” that are responsible for providing various functionalities to the initrd image that user can configure via configuration file. Although by default there’s no module that provides SSH server, dracut-sshd provides one for us.

Installation & configuration

Examples assumes that the server address is server.lan. Module installation and configuration is based on project’s documentation.

In order to work, dracut-sshd module needs an SSH authorized key to allow you to connect to the initrd remotely. Module will look for it in two locations:

• /root/.ssh/dracut_authorized_keys
• /root/.ssh/authorized_keys

In this example, I’ll use default authorized keys location (/root/.ssh/authorized_keys). Key generation process is straightforward - execute following commands on your local machine that will be connecting to the server via SSH:

# Generate public / private key pair
ssh-keygen -t ed25519 -a 100 -f ~/.ssh/id_ed25519_server.lan
# Copy public key to the server
ssh-copy-id -i ~/.ssh/id_ed25519_server.lan root@server.lan

For convenience configure a Host entry for the server in SSH client configuration - edit a file under ~/.ssh/config and add following section:

Host server
    HostName             server.lan
    User                 root
    IdentitiesOnly       yes
    IdentityFile         ~/.ssh/id_ed25519_server.lan

Now in order to connect to the server, you can simply execute ssh server. Next, you need to install dracut-sshd module. You can install it either manually (install instructions are in the project’s repository) or from COPR repository - I decided to go with latter because it is automatically updated with the system and pull all necessary dependencies:

dnf copr enable gsauthof/dracut-sshd
dnf install -y dracut-sshd

dracut-sshd module automatically installs a dependency package, dracut-network, that provides network support inside initrd image. Although the network module is installed, it’s not configured to setup a network connection at boot - in order to fix that, you need to create a networkd configuration file under /etc/systemd/network/20-wired.network that will be used to get a valid IP for the machine by DHCP:

[Match]
Name=e*

[Network]
DHCP=ipv4

The last configuration step is to tell dracut which files and modules needs to be included in the initrd image. To do that, create a separate configuration file at /etc/dracut.conf.d/90-networkd.conf:

install_items+=" /etc/systemd/network/20-wired.network "
add_dracutmodules+=" systemd-networkd "

Since current initrd image does not contain sshd module, you need to regenerate it with dracut -f -v. Once executed, you can validate in the output that systemd-networkd and sshd modules are included:

...
dracut: *** Including module: systemd-networkd ***
...
dracut: *** Including module: sshd ***
...

Once initrd image is successfully re-generated, you can reboot the server and test the SSH connection while it’s waiting for a password. For convenience, dracut-sshd appends .bash_history with a systemd-tty-ask-password-agent command, so you don’t have to type it in - just press “up arrow” key and confirm with an enter - it’ll then ask you for a password:

ssh server
-sh-5.0#
-sh-5.0# systemd-tty-ask-password-agent
Please enter passphrase for disk luks-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: ***********************
-sh-5.0#
Connection to server.lan closed by remote host.
Connection to server.lan closed.

If disks were successfully decrypted, initrd proceeds with booting process - that’s why the SSH connection is dropped (SSH server is shut down and systemd takes over). After a few seconds you can repeat ssh server, this time to log in into the booted server.

Security considerations

As server’s initrd allows logging in via SSH, you need to keep in mind what it means - if private key is leaked, your server will be compromised. Attacker would gain either access to root account or ability to modify a boot process to allow further attacks. That’s why it is very important to keep server’s SSH private key file safe.