How to Install RustDesk Remote Desktop Server on Ubuntu

RustDesk is an open-source remote desktop solution that gives you full control of your remote connections through a self-hosted server. It is an alternative to closed-source applications such as TeamViewer and AnyDesk, with a lightweight interface and end-to-end encrypted sessions that route through infrastructure you own.

This article explains how to install the RustDesk server on Ubuntu using Docker Compose. It covers deploying the ID and relay services, opening the ports that RustDesk clients require, retrieving the server encryption key, and connecting a client to your own server.

Prerequisites

Before you begin, you need to:

• Have access to an Ubuntu server as a non-root user with sudo privileges.
• Install Docker and Docker Compose.
• Point a subdomain such as rustdesk.example.com to your server's public IP address.

Understand the RustDesk Architecture

The RustDesk server runs as two services. hbbs is the ID and rendezvous server that registers clients and brokers connections between them. hbbr is the relay server that carries session traffic when two clients cannot reach each other directly.

Both services use their own binary protocols rather than HTTP, so they cannot be placed behind a standard HTTP reverse proxy. Clients connect to the ports directly, which means the firewall must permit them.

Install RustDesk

Docker Compose runs both services from a single file and restarts them automatically after a reboot. The compose file also maps a local directory into the containers so that the generated encryption key and the ID database survive container replacement.

1. Create a directory to hold the RustDesk files.

   CONSOLE

   Copy

   $ sudo mkdir -p /opt/rustdesk

   Explain Code

   Powered by Vultr AgentBeta

2. Change to the directory.

   CONSOLE

   Copy

   $ cd /opt/rustdesk

   Explain Code

   Powered by Vultr AgentBeta

3. Create the compose file.

   CONSOLE

   Copy

   $ sudo nano rustdesk.yml

   Explain Code

   Powered by Vultr AgentBeta

4. Add the following configuration. Replace rustdesk.example.com with your subdomain or your server's public IP address.

   YAML

   Copy

   networks:
     rustdesk-net:
       external: false
   
   services:
     hbbs:
       container_name: hbbs
       ports:
         - 21115:21115
         - 21116:21116
         - 21116:21116/udp
         - 21118:21118
       image: rustdesk/rustdesk-server:latest
       command: hbbs -r rustdesk.example.com:21117
       volumes:
         - ./data:/root
       networks:
         - rustdesk-net
       depends_on:
         - hbbr
       restart: unless-stopped
   
     hbbr:
       container_name: hbbr
       ports:
         - 21117:21117
         - 21119:21119
       image: rustdesk/rustdesk-server:latest
       command: hbbr
       volumes:
         - ./data:/root
       networks:
         - rustdesk-net
       restart: unless-stopped

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   • command: hbbs -r rustdesk.example.com:21117: Tells the ID server which relay address to hand out to clients. This value must be an address that clients can reach, so a loopback address such as 127.0.0.1 prevents relayed sessions from working.
   • volumes: ./data:/root: Persists the generated key pair and the ID database in /opt/rustdesk/data so that clients keep working after the containers are recreated.
   • restart: unless-stopped: Restarts both services automatically after a reboot.

5. Verify that the compose file parses correctly.

   CONSOLE

   Copy

   $ sudo docker compose -f rustdesk.yml config

   Explain Code

   Powered by Vultr AgentBeta

   The output displays the resolved configuration. A YAML error here means the indentation does not match the file above.

6. Start the services.

   CONSOLE

   Copy

   $ sudo docker compose -f rustdesk.yml up -d

   Explain Code

   Powered by Vultr AgentBeta

7. Verify that both containers are running.

   CONSOLE

   Copy

   $ sudo docker ps

   Explain Code

   Powered by Vultr AgentBeta

   The output displays the hbbs and hbbr containers with an Up status and their mapped ports.

Open the Firewall Ports

RustDesk clients connect to the server ports directly, so each one must be permitted through the firewall. Missing the UDP mapping on port 21116 is a common cause of clients that never come online, because that port carries the registration heartbeat.

1. Allow SSH so that enabling the firewall does not end your session.

   CONSOLE

   Copy

   $ sudo ufw allow OpenSSH

   Explain Code

   Powered by Vultr AgentBeta

2. Allow the ID server ports.

   CONSOLE

   Copy

   $ sudo ufw allow 21115/tcp

   Explain Code

   Powered by Vultr AgentBeta

3. Allow the registration port over both protocols.

   CONSOLE

   Copy

   $ sudo ufw allow 21116/tcp && sudo ufw allow 21116/udp

   Explain Code

   Powered by Vultr AgentBeta

4. Allow the relay port.

   CONSOLE

   Copy

   $ sudo ufw allow 21117/tcp

   Explain Code

   Powered by Vultr AgentBeta

5. Allow the web client ports.

   CONSOLE

   Copy

   $ sudo ufw allow 21118/tcp && sudo ufw allow 21119/tcp

   Explain Code

   Powered by Vultr AgentBeta

6. Enable the firewall.

   CONSOLE

   Copy

   $ sudo ufw enable

   Explain Code

   Powered by Vultr AgentBeta

   The command warns that it may disrupt existing SSH connections and asks for confirmation. Enter y to proceed.

7. Review the active rules.

   CONSOLE

   Copy

   $ sudo ufw status

   Explain Code

   Powered by Vultr AgentBeta

   The output displays OpenSSH and each RustDesk port with an ALLOW action.

Retrieve the Server Encryption Key

The ID server generates an Ed25519 key pair on first start and stores it in the mounted data directory. Clients must present the matching public key to register, which prevents unknown clients from using your server.

1. Display the public key.

   CONSOLE

   Copy

   $ sudo cat /opt/rustdesk/data/id_ed25519.pub

   Explain Code

   Powered by Vultr AgentBeta

   The output displays a single base64 string that ends with an equals sign.

2. Copy the value. Every client that connects to this server needs it.

   Note

   Keep the private key in /opt/rustdesk/data/id_ed25519 on the server. Deleting the data directory regenerates the key pair, and every client must then be reconfigured with the new public key.

Configure a RustDesk Client

Each client stores the server address and public key in its network settings. Repeat these steps on every machine you want to reach, including the machine you connect from.

1. Download the RustDesk client for your operating system from the official website.

2. Install and open the application.

3. Click the menu button next to your connection ID, then select Settings.

4. Select Network, then unlock the ID/Relay Server section.

5. Enter your subdomain or server IP address in the ID Server field.

6. Paste the server public key into the Key field.

7. Click OK to save the settings.

   The connection indicator turns green once the client registers with your server.

8. Note the ID and one-time password shown on the machine you want to control. To set a permanent password instead, open Settings, then Security, and set an unattended access password.

Test the Connection

Confirming a session end to end proves that both the ID server and the relay are reachable, and that the key matches on both sides.

1. Configure a second machine with the same ID Server and Key values.

2. Enter the remote machine's ID in the Control Remote Desktop field.

3. Click Connect.

4. Enter the remote machine's password when prompted.

   The remote desktop appears. Traffic now routes through your own server rather than the public RustDesk infrastructure.

Conclusion

You have installed a self-hosted RustDesk server on Ubuntu with Docker Compose, opened the ports that clients require, and connected a client using the server's encryption key. Because both services speak their own protocols rather than HTTP, keep the ports reachable rather than placing the server behind an HTTP reverse proxy. For more information, visit the official RustDesk documentation.