How to Deploy Redpanda – Kafka-Compatible Streaming Platform

Redpanda is a modern, high-performance streaming platform that supports the Kafka API while using a simpler, more efficient architecture. It runs natively in C++ and eliminates JVM components like ZooKeeper, so you avoid common Kafka maintenance difficulties like memory tuning, GC delays, and coordination overhead. This simplified design reduces latency, operating burden, and increases throughput.

This article shows how to set up and configure Redpanda on Ubuntu 24.04, and make its console accessible over a web interface with HTTPS. It covers system preparation, secure configuration practices, and the use of Redpanda tooling such as bootstrap configuration and rpk profiles.

Prerequisites

Before starting, ensure you:

• Have access to an Ubuntu 24.04 server as a non-root sudo user. The server should meet the memory and CPU requirements as per your use case.
• Configure a domain name, such as redpanda.example.com, to point to your server’s public IP address.

Install Redpanda

Redpanda provides a setup script that detects the operating system and configures the apt repositories automatically. Follow the steps below to update the APT package manager index and install Redpanda on your server.

1. Update the APT package index.

   CONSOLE

   Copy

   $ sudo apt update

   Explain Code

   Powered by Vultr AgentBeta

2. Download and execute the official Redpanda's APT setup script as root.

   CONSOLE

   Copy

   $ curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash

   Explain Code

   Powered by Vultr AgentBeta

   Warning

   If the script or its hosting source is compromised, it can run arbitrary commands and fully take over the system. Only run it if you trust the source, and your network.

3. Install the Redpanda server binary, the Redpanda tuner, and the Redpanda Keeper (rpk) CLI tool.

   CONSOLE

   Copy

   $ sudo apt install redpanda -y

   Explain Code

   Powered by Vultr AgentBeta

4. Verify the installation by checking the rpk version.

   CONSOLE

   Copy

   $ rpk --version

   Explain Code

   Powered by Vultr AgentBeta

Configure Firewall Access

By default, Redpanda services communicate on specific TCP ports. To allow external clients, administrators, and other nodes in the cluster to connect, you must explicitly let traffic through the Uncomplicated Firewall (UFW).

The table below outlines the specific ports used by Redpanda and their functions:

Follow the steps below to open the ports for each specific Redpanda component.

1. Open all required ports.

   CONSOLE

   Copy

   $ sudo ufw allow 9092,8082,8081,9644,33145/tcp

   Explain Code

   Powered by Vultr AgentBeta

2. To generate and renew TLS certificates with Let's Encrypt, Certbot requires port 80 to be open to perform ACME HTTP-01 challenge validation. Allow port 80 through your firewall.

   CONSOLE

   Copy

   $ sudo ufw allow 80/tcp

   Explain Code

   Powered by Vultr AgentBeta

3. Allow port 443 through your firewall to access Redpanda web console over HTTPS.

   CONSOLE

   Copy

   $ sudo ufw allow 443/tcp

   Explain Code

   Powered by Vultr AgentBeta

4. Reload the firewall.

   CONSOLE

   Copy

   $ sudo ufw reload

   Explain Code

   Powered by Vultr AgentBeta

Generate TLS Certificates with Let's Encrypt

By default, Redpanda data is sent unencrypted. This poses a significant security risk in production environments, as sensitive data streams could be intercepted. To secure your cluster, you must enable TLS encryption.

Follow the steps below to install Certbot, get a certificate, and organize the files for Redpanda.

1. Install Certbot.

   CONSOLE

   Copy

   $ sudo apt install certbot -y

   Explain Code

   Powered by Vultr AgentBeta

2. Set a shell variable for your domain name. Replace redpanda.example.com with your domain name.

   CONSOLE

   Copy

   $ DOMAIN=redpanda.example.com

   Explain Code

   Powered by Vultr AgentBeta

3. Set a shell variable for your email address. Replace admin@example.com with your email address.

   CONSOLE

   Copy

   $ EMAIL=admin@example.com

   Explain Code

   Powered by Vultr AgentBeta

4. Request a TLS certificate for your domain using Certbot.

   CONSOLE

   Copy

   $ sudo certbot certonly --standalone -d $DOMAIN --non-interactive --email $EMAIL

   Explain Code

   Powered by Vultr AgentBeta
   • The above command temporarily starts a standalone web server on port 80 to verify domain ownership with Let’s Encrypt.
   • Certbot stores certificates under /etc/letsencrypt/live, which is readable only by the root user.
   • Since Redpanda runs as the redpanda user, you must copy the certificates to a dedicated directory with appropriate permissions.

5. Create a directory to store Redpanda TLS certificates.

   CONSOLE

   Copy

   $ sudo mkdir /etc/redpanda/certs

   Explain Code

   Powered by Vultr AgentBeta

6. Copy the public certificate to the Redpanda certificate directory.

   CONSOLE

   Copy

   $ sudo cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/redpanda/certs/node.crt

   Explain Code

   Powered by Vultr AgentBeta

7. Copy the private key to the Redpanda certificate directory.

   CONSOLE

   Copy

   $ sudo cp /etc/letsencrypt/live/$DOMAIN/privkey.pem /etc/redpanda/certs/node.key

   Explain Code

   Powered by Vultr AgentBeta

8. Copy the CA certificate to the Redpanda certificate directory.

   CONSOLE

   Copy

   $ sudo cp /etc/letsencrypt/live/$DOMAIN/chain.pem /etc/redpanda/certs/ca.crt

   Explain Code

   Powered by Vultr AgentBeta

9. Grant the redpanda user ownership of the certificate directory.

   CONSOLE

   Copy

   $ sudo chown -R redpanda:redpanda /etc/redpanda/certs/

   Explain Code

   Powered by Vultr AgentBeta

10. Set restrictive permissions on the private key so only the owner can read it.

    CONSOLE

    Copy

    $ sudo chmod 400 /etc/redpanda/certs/node.key

    Explain Code

    Powered by Vultr AgentBeta

11. Allow read-only access to the public certificate and CA certificate.

    CONSOLE

    Copy

    $ sudo chmod 444 /etc/redpanda/certs/node.crt /etc/redpanda/certs/ca.crt

    Explain Code

    Powered by Vultr AgentBeta

Set Up and Verify Automatic Certificate Renewal

Let’s Encrypt certificates expire every 90 days. To avoid service outages, Certbot includes an automated renewal system using certbot.timer. You must add a deploy hook so that whenever certificates are renewed, your Redpanda TLS files are updated and the service reloads securely.

1. Create the renewal script.

   CONSOLE

   Copy

   $ sudo nano /etc/letsencrypt/renewal-hooks/deploy/redpanda-renewal.sh

   Explain Code

   Powered by Vultr AgentBeta

2. Add the following contents to the file. Replace redpanda.example.com with your domain.

   INI

   Copy

   #!/bin/bash
   DOMAIN="redpanda.example.com"
   
   # Copy renewed certificates
   cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/redpanda/certs/node.crt
   cp /etc/letsencrypt/live/$DOMAIN/privkey.pem /etc/redpanda/certs/node.key
   cp /etc/letsencrypt/live/$DOMAIN/chain.pem /etc/redpanda/certs/ca.crt
   
   # Set ownership and permissions
   chown -R redpanda:redpanda /etc/redpanda/certs/
   chmod 400 /etc/redpanda/certs/node.key
   chmod 444 /etc/redpanda/certs/node.crt /etc/redpanda/certs/ca.crt
   
   # Restart Redpanda to pick up new certificates
   systemctl restart redpanda

   Explain Code

   Powered by Vultr AgentBeta

3. Make the renewal script executable.

   CONSOLE

   Copy

   $ sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/redpanda-renewal.sh

   Explain Code

   Powered by Vultr AgentBeta

4. Run a dry-run renewal to verify that automatic renewal and the deploy hook work correctly.

   CONSOLE

   Copy

   $ sudo certbot renew --dry-run

   Explain Code

   Powered by Vultr AgentBeta

Configure Redpanda for Production

Redpanda runs in development mode by default, which disables hardware optimization and leaves the cluster open. To prepare for production, you will enable hardware tuning, then use the Bootstrap File method to initialize security settings and create the superuser before the cluster starts processing traffic.

Enable Production Mode and Tuning

The autotuner identifies your hardware configuration and optimizes the Linux kernel (I/O schedulers, CPU settings) for performance.

1. Set Redpanda to production mode.

   CONSOLE

   Copy

   $ sudo rpk redpanda mode production

   Explain Code

   Powered by Vultr AgentBeta

2. Tune the Linux kernel.

   CONSOLE

   Copy

   $ sudo rpk redpanda tune all

   Explain Code

   Powered by Vultr AgentBeta

   This takes about 30 seconds and dramatically increases performance. Changes to the kernel do not always persist across reboots. Enabling the redpanda-tuner service guarantees that optimizations are automatically reapplied when the node restarts.

3. Enable the Redpanda tuner service to persist optimizations across reboots.

   CONSOLE

   Copy

   $ sudo systemctl start redpanda-tuner

   Explain Code

   Powered by Vultr AgentBeta

Configure Bootstrap File and User

Redpanda allows you to configure cluster settings before the first startup using a .bootstrap.yaml file. This is the cleanest way to set up authentication and security from the beginning.

1. Create the Redpanda bootstrap configuration file.

   CONSOLE

   Copy

   $ sudo nano /etc/redpanda/.bootstrap.yaml

   Explain Code

   Powered by Vultr AgentBeta

2. Add the following initial configuration to the file.

   YAML

   Copy

   # Enable SASL authentication globally
   enable_sasl: true
   
   # Enable Admin API authentication
   admin_api_require_auth: true
   
   # Define superusers (users with full administrative privileges)
   superusers:
     - admin

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file. This configuration ensures authentication and administrative access are enforced from the very first startup.

   Note

   This file is only read on the first startup of Redpanda. Any later configuration changes must be done through the Admin API or rpk cluster config commands.

   The bootstrap file defines who the superuser is, but it does not create the password. To create the user credentials on the first boot, define an environment variable in the Redpanda system configuration.

3. Add the RP_BOOTSTRAP_USER variable to the bottom of the file. Replace STRONG_PASSWORD with your actual password.

   CONSOLE

   Copy

   $ echo "RP_BOOTSTRAP_USER=admin:STRONG_PASSWORD:SCRAM-SHA-256" | sudo tee -a /etc/default/redpanda

   Explain Code

   Powered by Vultr AgentBeta

   This creates a user named admin with your specified password using the SCRAM-SHA-256 authentication mechanism.

Update Configuration File

Redpanda uses a YAML configuration file located at /etc/redpanda/redpanda.yaml. By default, the service listens on the localhost interface. To allow external connections, configure the advertised addresses. Now configure the main Redpanda settings with your domain name and TLS certificates.

1. Back up the original configuration.

   CONSOLE

   Copy

   $ sudo cp /etc/redpanda/redpanda.yaml /etc/redpanda/redpanda-backup.yaml

   Explain Code

   Powered by Vultr AgentBeta

2. Open the configuration file.

   CONSOLE

   Copy

   $ sudo nano /etc/redpanda/redpanda.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Replace the entire configuration file with the following content. Replace redpanda.example.com with your actual domain name.

   YAML

   Copy

   redpanda:
     data_directory: /var/lib/redpanda/data
     seed_servers: []
   
     rpc_server:
       address: 0.0.0.0
       port: 33145
     advertised_rpc_api:
       address: redpanda.example.com
       port: 33145
   
     kafka_api:
       - name: tls_listener
         address: 0.0.0.0
         port: 9092
         authentication_method: sasl
     advertised_kafka_api:
       - name: tls_listener
         address: redpanda.example.com
         port: 9092
     kafka_api_tls:
       - name: tls_listener
         key_file: /etc/redpanda/certs/node.key
         cert_file: /etc/redpanda/certs/node.crt
         truststore_file: /etc/redpanda/certs/ca.crt
         enabled: true
         require_client_auth: false
   
     admin:
       - address: 0.0.0.0
         port: 9644
     admin_api_tls:
       - enabled: true
         key_file: /etc/redpanda/certs/node.key
         cert_file: /etc/redpanda/certs/node.crt
         truststore_file: /etc/redpanda/certs/ca.crt
   
   rpk:
     tune_network: true
     tune_disk_scheduler: true
     tune_disk_nomerges: true
     tune_disk_write_cache: true
     tune_disk_irq: true
     tune_cpu: true
     tune_aio_events: true
     tune_clocksource: true
     tune_swappiness: true
     coredump_dir: /var/lib/redpanda/coredump
     tune_ballast_file: true

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file. This configuration defines the core Redpanda node settings:

   redpanda

   • Configures the local data directory and disables seed servers for a single-node cluster.
   • Defines the RPC API used for internal node communication and advertises it using the public hostname.
   • Enables the Kafka API with SASL authentication over TLS on port 9092.
   • Advertises the Kafka endpoint using the public domain so clients can connect securely.
   • Configures TLS certificates for Kafka traffic using the node certificate and cluster CA.

   admin API

   • Exposes the Redpanda Admin API on port 9644.
   • Secures the Admin API with TLS using the same certificate authority.
   • Ensures administrative operations require encrypted connections.

   rpk tuning

   • Applies recommended system tuning for network, disk, CPU, memory, and I/O.
   • Improves performance and stability for production workloads.
   • Configures a dedicated directory for Redpanda core dumps.

4. Start the Redpanda service.

   CONSOLE

   Copy

   $ sudo systemctl start redpanda

   Explain Code

   Powered by Vultr AgentBeta

   Redpanda initializes the node, applies the bootstrap configuration, creates the configured superuser, and brings the cluster online with authentication and TLS enforced.

5. Verify the service is active.

   CONSOLE

   Copy

   $ sudo systemctl status redpanda

   Explain Code

   Powered by Vultr AgentBeta

Create the Profile Configuration File

Run rpk as your regular sudo-capable user (not via sudo) so it can use your profile configuration.

1. Create a new rpk profile file to store your Redpanda credentials.

   CONSOLE

   Copy

   $ nano ~/rpk-profile.yaml

   Explain Code

   Powered by Vultr AgentBeta

2. Add the following configuration to the file. Replace redpanda.example.com and STRONG_PASSWORD with your actual broker address and password.

   YAML

   Copy

   kafka_api:
     sasl:
       user: admin
       password: STRONG_PASSWORD
       mechanism: SCRAM-SHA-256
     tls:
       enabled: true
       ca_file: /etc/redpanda/certs/ca.crt
     brokers:
       - redpanda.example.com:9092
   
   admin_api:
     tls:
       enabled: true
       ca_file: /etc/redpanda/certs/ca.crt
     addresses:
       - redpanda.example.com:9644

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

3. Create and switch to the profile from the file.

   CONSOLE

   Copy

   $ rpk profile create production --from-profile ~/rpk-profile.yaml

   Explain Code

   Powered by Vultr AgentBeta

4. Now check the cluster health. Thanks to the rpk profile, you don't need to specify credentials or TLS flags.

   CONSOLE

   Copy

   $ rpk cluster health

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   CLUSTER HEALTH OVERVIEW
   =======================
   Healthy:                          true
   Unhealthy reasons:                []
   Controller ID:                    0
   All nodes:                        [0]
   Nodes down:                       []
   Nodes in recovery mode:           []
   Leaderless partitions (0):        []
   Under-replicated partitions (0):  []
   Cluster UUID:                     65e28fbb-c1b4-42c3-b0b7-fd82608699f3

Test the Installation

Now that the cluster is secured and running, perform a basic producer-consumer test to ensure it processes data accurately.

1. Create a test topic named test-topic.

   CONSOLE

   Copy

   $ rpk topic create test-topic

   Explain Code

   Powered by Vultr AgentBeta

   Output should show:

   TOPIC       STATUS
   test-topic  OK

2. Produce a message to the topic.

   CONSOLE

   Copy

   $ echo "Hello Redpanda" | rpk topic produce test-topic

   Explain Code

   Powered by Vultr AgentBeta

   Output should show:

   Produced to partition 0 at offset 0 with timestamp 1699618845123.

3. Consume the message from the topic.

   CONSOLE

   Copy

   $ rpk topic consume test-topic --num 1

   Explain Code

   Powered by Vultr AgentBeta

   The output should display the message payload and metadata in JSON format.

Access Redpanda Console

Redpanda Console is a modern web interface for managing and monitoring your cluster. It provides visibility into topics, messages, consumer groups, and the schema registry.

1. Having configured the Redpanda repository earlier, install the console package.

   CONSOLE

   Copy

   $ sudo apt install redpanda-console -y

   Explain Code

   Powered by Vultr AgentBeta

2. Configure the Redpanda Console configuration file.

   CONSOLE

   Copy

   $ sudo nano /etc/redpanda/redpanda-console-config.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Replace the existing content with the following configuration.

   YAML

   Copy

   server:
     listenPort: 8080
     listenAddress: 127.0.0.1
   
   kafka:
     brokers:
       - redpanda.example.com:9092
     tls:
       enabled: true
       caFilepath: /etc/redpanda/certs/ca.crt
     sasl:
       enabled: true
       username: admin
       password: STRONG_PASSWORD
       mechanism: SCRAM-SHA-256
   
   redpanda:
     adminApi:
       enabled: true
       urls:
         - https://redpanda.example.com:9644
       tls:
         enabled: true
         caFilepath: /etc/redpanda/certs/ca.crt
       authentication:
         basic:
           username: admin
           password: STRONG_PASSWORD

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The redpanda-console-config.yaml file controls how the console connects to your Redpanda cluster.

   • Server: Sets the internal port (8080) where the console listens.
   • Kafka & Redpanda: Establishes secure TLS connections to your cluster's Kafka API (messaging) and Admin API (management) using the credentials defined earlier.

4. Enable the Redpanda Console service to start automatically on boot.

   CONSOLE

   Copy

   $ sudo systemctl enable redpanda-console

   Explain Code

   Powered by Vultr AgentBeta

5. Start the Redpanda Console service.

   CONSOLE

   Copy

   $ sudo systemctl start redpanda-console

   Explain Code

   Powered by Vultr AgentBeta

Set Up Nginx Reverse Proxy with Basic Authentication

Since the open-source version of Redpanda Console does not include built-in login functionality, you must secure it using Nginx Basic Authentication. This setup forces users to enter a username and password before they can access the dashboard.

1. Install Nginx.

   CONSOLE

   Copy

   $ sudo apt install nginx -y

   Explain Code

   Powered by Vultr AgentBeta

2. Install the Apache utilities package.

   CONSOLE

   Copy

   $ sudo apt install apache2-utils -y

   Explain Code

   Powered by Vultr AgentBeta

   This package provides the htpasswd utility, which is required to generate the password file for Nginx.

3. Run the following command to create a user named admin. You will be prompted to enter and confirm a password.

   CONSOLE

   Copy

   $ sudo htpasswd -c /etc/nginx/.htpasswd admin

   Explain Code

   Powered by Vultr AgentBeta

   The above command prompts you to enter and confirm the password for the admin user.

4. Create a new Nginx configuration file for the console.

   CONSOLE

   Copy

   $ sudo nano /etc/nginx/sites-available/redpanda-console

   Explain Code

   Powered by Vultr AgentBeta

5. Add the following configuration. Replace redpanda.example.com with your domain.

   INI

   Copy

   server {
       listen 80;
       server_name redpanda.example.com;
       return 301 https://$host$request_uri;
   }
   
   server {
       listen 443 ssl;
       server_name redpanda.example.com;
   
       # Reuse the existing Let's Encrypt certificates
       ssl_certificate /etc/letsencrypt/live/redpanda.example.com/fullchain.pem;
       ssl_certificate_key /etc/letsencrypt/live/redpanda.example.com/privkey.pem;
   
       location / {
           auth_basic "Restricted Access";
           auth_basic_user_file /etc/nginx/.htpasswd;
           proxy_pass http://127.0.0.1:8080;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       }
   }

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

6. Enable the new site configuration.

   CONSOLE

   Copy

   $ sudo ln -s /etc/nginx/sites-available/redpanda-console /etc/nginx/sites-enabled/

   Explain Code

   Powered by Vultr AgentBeta

7. Test the Nginx configuration for syntax errors.

   CONSOLE

   Copy

   $ sudo nginx -t

   Explain Code

   Powered by Vultr AgentBeta

8. Restart Nginx to apply the changes.

   CONSOLE

   Copy

   $ sudo systemctl restart nginx

   Explain Code

   Powered by Vultr AgentBeta

9. Open your web browser and navigate to your domain using HTTPS.

   https://redpanda.example.com

   The browser prompts you to enter the username and the password you created earlier. After entering your credentials, you should see the Redpanda Console dashboard secured with a valid TLS certificate.

   

Conclusion

In this article, you installed a single-binary Redpanda cluster on Ubuntu 24.04, secured external access using TLS encryption with Let's Encrypt, configured automatic certificate renewal, and set up Redpanda Console for observability. For more information, refer to the Redpanda Documentation.