How to Deploy Lustre as an AWS FSx Alternative

Lustre is an open-source, POSIX-compliant parallel distributed file system designed for high-performance computing (HPC), machine learning training, and large-scale data processing. It separates file metadata from file data and distributes each across dedicated server nodes, which delivers aggregate throughput that scales linearly as you add storage targets. AWS FSx for Lustre is a fully managed Lustre file system that offers SSD and Intelligent-Tiering storage classes, native S3 integration, per-client throughput up to 1,200 Gbps, and automatic scaling, but it applies per-TiB pricing and ties workloads to the AWS ecosystem.

This article explains how to deploy Lustre as an alternative to AWS FSx for Lustre. It covers the Lustre package installation, metadata and object storage server deployment, client mounts, striping and Progressive File Layouts (PFL), LNET networking, quotas and POSIX access control lists (ACLs), monitoring with Prometheus and Grafana, high availability with Pacemaker, benchmarking with IOR and mdtest, and migration from AWS FSx for Lustre.

Understanding Lustre Architecture

Lustre separates file metadata from file data and distributes each across dedicated servers. The Metadata Server (MDS) hosts the Metadata Target (MDT), which stores filenames, directory structure, permissions, and stripe layout information. Object Storage Server (OSS) nodes host Object Storage Targets (OSTs) that store the actual file data. Clients communicate directly with both the MDS and the OSS nodes through the LNET network protocol, so file data flows in parallel across multiple OSTs without passing through a central gateway.

The following table maps each AWS FSx for Lustre feature to its self-hosted Lustre equivalent:

AWS FSx Feature	Self-Hosted Lustre Equivalent
FSx for Lustre File System	Lustre file system (MDS and OSS cluster)
FSx Metadata	Lustre MDS (Metadata Server) and MDT (Metadata Target)
FSx Data Storage	Lustre OSS (Object Storage Servers) and OSTs (Object Storage Targets)
FSx SSD Storage Class	Lustre with LDISKFS backend on NVMe or SSD
FSx Intelligent-Tiering	Lustre HSM (Hierarchical Storage Management) and PFL (Progressive File Layout)
FSx S3 Data Repository	Lustre HSM with an S3-compatible object storage backend
FSx Auto Import/Export	Lustre Robinhood Policy Engine or `lhsmtool_posix`
FSx Throughput Scaling	Adding OSS and OST pairs
FSx Client Mount	Lustre kernel client module
CloudWatch Metrics	Lustre Jobstats and Prometheus exporter

Prerequisites

Before you begin, you need to:

Have access to at least three Linux-based servers as a non-root user with sudo privileges: one Metadata Server (MDS) and two Object Storage Servers (OSS).
Have access to a separate Linux-based server as a non-root user with sudo privileges to act as the client that mounts the file system.
Attach a dedicated block storage volume to the MDS for the Metadata Target (at least 10 GB) and to each OSS node for an Object Storage Target (at least 20 GB per OSS).
Connect all nodes over a private VPC for the Lustre network (10 GbE minimum; InfiniBand is recommended for production HPC workloads).
Have basic Linux storage administration knowledge, including kernel module loading and matching kernel-devel packages to the running kernel.

Install Lustre Packages

Lustre 2.15 LTS requires a kernel that the Whamcloud project patched and rebuilt with Lustre-specific changes. Whamcloud publishes pre-built packages for Enterprise Linux 8 and 9 and for Ubuntu. This article installs the Enterprise Linux 8 build, which runs on Rocky Linux 8, AlmaLinux 8, and RHEL 8, and pulls the patched kernel and modules directly from the Whamcloud repository. For a different distribution, select the matching repository from the Whamcloud download site and adjust the package names accordingly.

Add the Lustre Repository

The Whamcloud repository serves the Lustre server packages, the matching client packages, and the patched e2fsprogs package that provides Lustre-specific file system utilities.

Open a repository configuration file on each MDS, OSS, and client node.
console
```
$ sudo nano /etc/yum.repos.d/lustre.repo
```

Add the following configuration.

                            ini
                            
                        
[lustre-server]
name=Lustre Server
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.15.8/el8.10/server/
gpgcheck=0
enabled=1

[lustre-client]
name=Lustre Client
baseurl=https://downloads.whamcloud.com/public/lustre/lustre-2.15.8/el8.10/client/
gpgcheck=0
enabled=1

[e2fsprogs-wc]
name=Lustre e2fsprogs
baseurl=https://downloads.whamcloud.com/public/e2fsprogs/latest/el8/
gpgcheck=0
enabled=1

Save and close the file.

lustre-server: Provides the server-side packages, including the patched kernel, kernel modules, and management utilities.
lustre-client: Provides the client-side kernel modules and userspace utilities such as lfs.
e2fsprogs-wc: Provides the Whamcloud-built e2fsprogs package that supports the Lustre LDISKFS backend.

Install Server Packages

Run the following steps on the MDS and on both OSS nodes.

Install the EPEL repository, which provides several dependencies.
console
```
$ sudo dnf install -y epel-release
```

Install the Lustre-patched kernel and matching kernel headers.

                            console
                            
$ sudo dnf install -y kernel-4.18.0-553.82.1.el8_lustre kernel-devel-4.18.0-553.82.1.el8_lustre --nogpgcheck

Set the patched kernel as the default boot kernel.

                            console
                            
$ sudo grubby --set-default /boot/vmlinuz-4.18.0-553.82.1.el8_lustre.x86_64

Install the Whamcloud-built e2fsprogs.

                            console
                            
$ sudo dnf install -y e2fsprogs --enablerepo=e2fsprogs-wc --nogpgcheck

Install the Lustre server packages.

                            console
                            
$ sudo dnf install -y lustre-osd-ldiskfs-mount lustre kmod-lustre-osd-ldiskfs --nogpgcheck

Pin the kernel version to prevent automatic updates from breaking Lustre.
console
```
$ echo "exclude=kernel-4.18.0*" | sudo tee -a /etc/dnf/dnf.conf
```
Note
The kernel pin is essential. Lustre server modules build against a specific kernel version, and automatic updates from the regular distribution repositories install kernels that the Lustre modules cannot load against, which breaks the cluster on the next reboot.
Reboot the node into the patched kernel.
console
```
$ sudo reboot
```
Wait approximately 60 seconds for the node to reboot, then reconnect over SSH.
Confirm the running kernel after reconnecting.
console
```
$ uname -r
```
The output displays the Lustre-patched kernel.
```
4.18.0-553.82.1.el8_lustre.x86_64
```
Load the Lustre kernel module.
console
```
$ sudo modprobe lustre
```
Confirm that the modules load correctly.
console
```
$ sudo lsmod | grep lustre
```
The output lists the lustre, lnet, obdclass, and related modules.

Install Client Packages

Run the following steps on the client node only. The client uses the stock distribution kernel and does not require the patched kernel.

Install the EPEL repository.
console
```
$ sudo dnf install -y epel-release
```

Install the Lustre client packages.

                            console
                            
$ sudo dnf install -y kmod-lustre-client lustre-client --nogpgcheck

Load the Lustre kernel module.
console
```
$ sudo modprobe lustre
```
Verify that the client modules load.
console
```
$ sudo lsmod | grep lustre
```
Pin the kernel version to prevent automatic updates from breaking the client module.
console
```
$ echo "exclude=kernel-4.18.0*" | sudo tee -a /etc/dnf/dnf.conf
```

Open the Lustre Network Port

Lustre nodes communicate over LNET on TCP port 988. Enterprise Linux enables firewalld by default, which blocks this port and prevents the OSS nodes and clients from reaching the MGS. Complete the following steps on every MDS, OSS, and client node.

Allow the LNET port through the firewall.
console
```
$ sudo firewall-cmd --permanent --add-port=988/tcp
```
Reload the firewall to apply the rule.
console
```
$ sudo firewall-cmd --reload
```

Important

By default, LNET uses each server's primary network interface, which is the public interface on Vultr instances. The MGS registers its network identifier (NID) on that interface, so the MDS-IP-ADDRESS you use to format OSTs and mount clients must match it. To route Lustre traffic over the private VPC instead, configure LNET on every node before you format any target. Create /etc/modprobe.d/lustre.conf with options lnet networks=tcp0(INTERFACE), replacing INTERFACE with the VPC interface name from ip addr (for example, enp8s0), then reboot the node. Use the node's VPC address as MDS-IP-ADDRESS throughout this article.

Deploy the Metadata Server

The Metadata Server hosts the Management Server (MGS) and Metadata Target (MDT) roles in this deployment, which collocates them on the same node. The MGS coordinates file system registration for OSTs and clients, and the MDT stores the namespace and stripe layout information for the entire file system.

Identify the dedicated block device on the MDS.
console
```
$ lsblk
```
The output lists all attached disks. On Vultr, block storage uses virtio device names: the operating system occupies /dev/vda and the dedicated MDT volume appears as /dev/vdb. Use the device name shown in your own output in the commands that follow.
Format the device as the combined MGS and MDT. Replace lustrefs with your chosen file system name and /dev/vdb with your MDT device.
console
```
$ sudo mkfs.lustre --fsname=lustrefs --mgs --mdt --index=0 /dev/vdb
```
Create the mount point.
console
```
$ sudo mkdir -p /mnt/mdt
```

Mount the MDT.

                            console
                            
$ sudo mount -t lustre /dev/vdb /mnt/mdt

Verify that the MDS services are running.
console
```
$ sudo lctl dl
```
The output lists the active devices, including mgs, mds, mdt, and osd-ldiskfs.

Deploy Object Storage Servers

Each OSS hosts one OST that stores file data. The MGS coordinates the registration of OSTs with the file system, and clients communicate directly with each OSS for I/O. This deployment provisions one OST per OSS node with sequential indexes that the MGS uses to distinguish each target.

Configure the First OSS

Run the following steps on the first OSS node.

Identify the dedicated block device.
console
```
$ lsblk
```
Format the device as an OST that points to the MGS. Replace MDS-IP-ADDRESS with the MDS address on the Lustre network and /dev/vdb with your OST device.
console
```
$ sudo mkfs.lustre --fsname=lustrefs --ost --index=0 --mgsnode=MDS-IP-ADDRESS@tcp /dev/vdb
```
Create the mount point.
console
```
$ sudo mkdir -p /mnt/ost0
```

Mount the OST.

                            console
                            
$ sudo mount -t lustre /dev/vdb /mnt/ost0

Verify the OSS services.
console
```
$ sudo lctl dl
```
The output lists obdfilter, ost, and the connection to the MGS through the mgc device.

Configure the Second OSS

Run the following steps on the second OSS node. The second OSS uses index 1 so that the MGS distinguishes it from the first OST.

Format the device with index 1. Replace MDS-IP-ADDRESS with the MDS address on the Lustre network.
console
```
$ sudo mkfs.lustre --fsname=lustrefs --ost --index=1 --mgsnode=MDS-IP-ADDRESS@tcp /dev/vdb
```
Create the mount point for the second OST.
console
```
$ sudo mkdir -p /mnt/ost1
```

Mount the second OST.

                            console
                            
$ sudo mount -t lustre /dev/vdb /mnt/ost1

Switch to the MDS node and verify that both OSTs register successfully.

                            console
                            
$ sudo lctl get_param osc.*.ost_server_uuid

The output lists both OSTs in the FULL state.

osc.lustrefs-OST0000-osc-MDT0000.ost_server_uuid=lustrefs-OST0000_UUID  FULL
osc.lustrefs-OST0001-osc-MDT0000.ost_server_uuid=lustrefs-OST0001_UUID  FULL

Mount the Filesystem on Clients

The client connects to the file system through the MGS network address. After mounting, applications interact with the file system through standard POSIX system calls, so existing FSx workloads run without code changes.

Create the mount point on the client node.
console
```
$ sudo mkdir -p /mnt/lustre
```
Mount the file system. Replace MDS-IP-ADDRESS with the MDS address on the Lustre network and lustrefs with your file system name.
console
```
$ sudo mount -t lustre MDS-IP-ADDRESS@tcp:/lustrefs /mnt/lustre
```
Verify the mount and view the aggregate capacity.
console
```
$ df -h /mnt/lustre
```
View per-target capacity and usage.
console
```
$ lfs df -h /mnt/lustre
```
The output lists each MDT and OST separately, which helps identify imbalances and full targets.
Run a write test to confirm functionality.
console
```
$ sudo dd if=/dev/zero of=/mnt/lustre/testfile bs=1M count=100
```
The command reports the achieved write throughput.
Remove the test file.
console
```
$ sudo rm /mnt/lustre/testfile
```

Configure Striping and File Layouts

Striping distributes a single file across multiple OSTs, which delivers aggregate throughput beyond what a single storage target provides. Progressive File Layouts (PFL) apply different stripe counts to different file size ranges, which optimizes both small and large files in the same file system. PFL is the self-hosted equivalent of FSx Intelligent-Tiering at the layout level. Run the following steps on the client node.

Set a Default Stripe Layout

A uniform stripe configuration applies the same stripe count to every file in a directory.

Create a directory for striped files.
console
```
$ sudo mkdir -p /mnt/lustre/striped
```
Set the stripe count to 2 and the stripe size to 1 MiB.
console
```
$ sudo lfs setstripe -c 2 -S 1M /mnt/lustre/striped
```
- -c 2: Stripes each file across 2 OSTs.
- -S 1M: Sets the stripe size to 1 MiB, so Lustre writes 1 MiB to one OST before moving to the next.
Verify the stripe configuration.
console
```
$ sudo lfs getstripe -d /mnt/lustre/striped
```
The output reports stripe_count: 2 and stripe_size: 1048576.

Configure Progressive File Layouts

PFL adjusts the stripe count automatically based on file size, which improves performance for mixed workloads.

Create a directory for PFL files.
console
```
$ sudo mkdir -p /mnt/lustre/pfl
```
Apply a three-tier PFL.
console
```
$ sudo lfs setstripe -E 4M -c 1 -E 64M -c 2 -E -1 -c 2 /mnt/lustre/pfl
```
- -E 4M -c 1: Files up to 4 MiB use a single stripe, which avoids OST overhead for small files.
- -E 64M -c 2: Files up to 64 MiB use 2 stripes.
- -E -1 -c 2: Files larger than 64 MiB use 2 stripes for the remaining extents.
Confirm the PFL configuration.
console
```
$ sudo lfs getstripe -d /mnt/lustre/pfl
```
The output lists each tier with its extent boundaries and stripe count.

Set Up Networking with LNET

LNET is the network abstraction layer that Lustre uses for client-server and server-server communication. The default configuration uses TCP over the interface you set in the Open the Lustre Network Port section. Multi-rail configurations bond multiple interfaces for redundancy or aggregate bandwidth, while InfiniBand deployments use the o2ib transport instead of tcp. Run the following steps on the client node.

Display the current LNET configuration.
console
```
$ sudo lnetctl net show
```
The output shows the active networks, including the loopback (lo) and the primary TCP interface (tcp).
View detailed LNET tunables.
console
```
$ sudo lnetctl net show --verbose
```
The output exposes per-network statistics for sent and received packets along with the current peer credit configuration.
Test peer connectivity from the client to the MDS. Replace MDS-IP-ADDRESS with the MDS address on the Lustre network.
console
```
$ sudo lctl ping MDS-IP-ADDRESS@tcp
```
A successful response confirms that LNET routes traffic to the MDS correctly.

Configure Quotas and Access Control

Quotas enforce per-user and per-group capacity limits across the file system, while POSIX ACLs provide fine-grained file and directory permissions beyond the standard owner-group-other modes.

Enable Quotas

Quotas activate at the MDT and OST level, with separate enforcement for blocks (capacity) and inodes (file count).

Enable user and group quotas on the MDT from the MDS node.
console
```
$ sudo lctl conf_param lustrefs.quota.mdt=ug
```
Enable user and group quotas on the OSTs from the MDS node.
console
```
$ sudo lctl conf_param lustrefs.quota.ost=ug
```

Apply Quota Limits

Quota limits use soft and hard ceilings: the soft limit triggers a warning grace period, and the hard limit blocks further writes.

Set quota limits for a specific user from the client. Replace USER-NAME with the target Linux username.
console
```
$ sudo lfs setquota -u USER-NAME -b 100M -B 200M -i 10000 -I 20000 /mnt/lustre
```
- -b 100M: Soft block limit (100 MiB).
- -B 200M: Hard block limit (200 MiB).
- -i 10000: Soft inode limit (10,000 files).
- -I 20000: Hard inode limit (20,000 files).
View the quota usage and limits.
console
```
$ sudo lfs quota -u USER-NAME /mnt/lustre
```

Configure POSIX ACLs

POSIX ACLs grant fine-grained permissions to additional users or groups beyond the standard owner-group-other model. Run the following steps on the client node.

Create a directory for the ACL test.
console
```
$ sudo mkdir -p /mnt/lustre/acltest
```
Apply an ACL that grants read, write, and execute permissions to the nobody user.
console
```
$ sudo setfacl -m u:nobody:rwx /mnt/lustre/acltest
```
Verify the ACL.
console
```
$ sudo getfacl /mnt/lustre/acltest
```
The output lists the standard POSIX permissions and the additional user:nobody:rwx entry.

Set Up Monitoring

Lustre exposes detailed performance metrics through the lctl interface. The lustre_exporter translates those metrics into Prometheus format, and Grafana visualizes them through dashboards. The combination provides the self-hosted equivalent of CloudWatch metrics for FSx.

Enable Jobstats

Jobstats track per-job I/O statistics across the cluster, which makes it possible to attribute throughput and operation rates to specific applications.

Enable jobstats on the MDS.

                            console
                            
$ sudo lctl set_param jobid_var=procname_uid

Enable jobstats on the client.
console
```
$ sudo lctl set_param jobid_var=procname_uid
```
The procname_uid value tags each I/O operation with the originating process name and user ID, which produces job entries such as dd.0 for dd run by user 0 (root).

Build the Lustre Exporter

Run the following steps on the MDS. The Lustre exporter is a Go binary that the GSI-HPC community maintains.

Install the build dependencies.

                            console
                            
$ sudo dnf install -y golang git wget --nogpgcheck

Clone the exporter repository into your home directory.

                            console
                            
$ git clone https://github.com/GSI-HPC/lustre_exporter.git ~/lustre_exporter

Enter the cloned directory.
console
```
$ cd ~/lustre_exporter
```
Build the binary. The GOTOOLCHAIN=auto setting allows Go to download a newer toolchain if the source requires it.
console
```
$ GOTOOLCHAIN=auto go build
```
Verify that the binary exists.
console
```
$ ls -la lustre_exporter
```

Install the Exporter as a Service

A systemd service keeps the exporter running across reboots and restarts it on failure.

Copy the binary to the system path.

                            console
                            
$ sudo cp ~/lustre_exporter/lustre_exporter /usr/local/bin/

Create the systemd unit file.

                            console
                            
$ sudo nano /etc/systemd/system/lustre_exporter.service

Add the following content.

                            ini
                            
                        
[Unit]
Description=Lustre Prometheus Exporter
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/lustre_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

Save and close the file. The exporter listens on port 9169 by default.

Reload the systemd configuration.
console
```
$ sudo systemctl daemon-reload
```
Enable and start the exporter.
console
```
$ sudo systemctl enable --now lustre_exporter
```
Verify that the exporter serves Lustre metrics on port 9169.
console
```
$ curl -s http://localhost:9169/metrics | grep -E "^lustre_" | head -20
```
The output lists Lustre-specific metrics for capacity, LNET statistics, and component health.

Deploy Prometheus

Prometheus scrapes the metrics from the Lustre exporter and stores them as time-series data. Run the following steps on the MDS.

Download the Prometheus archive.

console

$ wget https://github.com/prometheus/prometheus/releases/download/v3.13.1/prometheus-3.13.1.linux-amd64.tar.gz

Extract the archive.

                            console
                            
$ tar -xzf prometheus-3.13.1.linux-amd64.tar.gz

Move the extracted directory to a system location.
console
```
$ sudo mv prometheus-3.13.1.linux-amd64 /opt/prometheus
```
Open the Prometheus configuration file.
console
```
$ sudo nano /opt/prometheus/prometheus.yml
```

Replace the contents with the following configuration.

                            yaml
                            
                        
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'lustre'
    static_configs:
      - targets: ['localhost:9169']

Save and close the file.

Create the systemd unit file for Prometheus.
console
```
$ sudo nano /etc/systemd/system/prometheus.service
```

Add the following content.

                            ini
                            
                        
[Unit]
Description=Prometheus
After=network.target

[Service]
Type=simple
ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/opt/prometheus/data
Restart=on-failure

[Install]
WantedBy=multi-user.target

Save and close the file.

Apply the SELinux context that permits systemd to execute the binary.
console
```
$ sudo chcon -t bin_t /opt/prometheus/prometheus
```
Reload the systemd configuration.
console
```
$ sudo systemctl daemon-reload
```
Enable and start Prometheus.
console
```
$ sudo systemctl enable --now prometheus
```
Verify that Prometheus scrapes the Lustre exporter successfully.
console
```
$ curl -s http://localhost:9090/api/v1/targets | grep '"health"'
```
The output reports "health":"up" for the Lustre target.

Install Grafana

Grafana provides interactive dashboards for the Lustre metrics that Prometheus stores. Run the following steps on the MDS.

Open the Grafana repository configuration.
console
```
$ sudo nano /etc/yum.repos.d/grafana.repo
```

Add the following content.

                            ini
                            
                        
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

Save and close the file.

Install Grafana.

                            console
                            
$ sudo dnf install -y grafana --nogpgcheck

Enable and start Grafana so it initializes the internal database with the correct ownership.
console
```
$ sudo systemctl enable --now grafana-server
```

Reset the admin password. Replace ADMIN-PASSWORD with a strong password.

                            console
                            
$ sudo grafana cli --homepath /usr/share/grafana admin reset-admin-password ADMIN-PASSWORD

Allow incoming traffic on port 3000 through the firewall.
console
```
$ sudo firewall-cmd --permanent --add-port=3000/tcp
```
Reload the firewall to apply the rule.
console
```
$ sudo firewall-cmd --reload
```
Open http://MDS-IP-ADDRESS:3000 in a web browser, replacing MDS-IP-ADDRESS with the public IP of your MDS node, then log in as admin with the password you set.
Click Connections, select Data sources, click Add data source, and choose Prometheus.
Set the connection URL to http://localhost:9090, then click Save & test. Grafana confirms successful connectivity to the Prometheus API.
Click Dashboards, select New, then New dashboard, and click Add visualization.
Choose Prometheus as the data source, switch the query editor to Code mode, and enter the following query: lustre_capacity_kibibytes.
Click Run queries to display MDT and MGS capacity over time, then click Save to persist the dashboard.

Configure High Availability

Pacemaker and Corosync provide active-passive failover for the MDS. The cluster monitors node health and migrates the MDT mount to a standby node when the active node fails. This section requires a second MDS node with the Lustre server packages installed, and a block device that both MDS nodes can access.

Important

Shared-storage failover requires a block device that both MDS nodes attach simultaneously through a SAN, iSCSI target, or a cloud multi-attach feature. Vultr Block Storage attaches to one instance at a time, so the shared MDT resource in this section is illustrative on Vultr. The cluster stack itself deploys and forms a quorum as shown, which you can pair with shared storage on infrastructure that provides it.

Install Cluster Packages

Run the following steps on both MDS nodes. The cluster software resides in the Enterprise Linux high availability repository.

Enable the high availability repository.
console
```
$ sudo dnf config-manager --set-enabled ha
```

Install Pacemaker, Corosync, pcs, and the fence agents.

                            console
                            
$ sudo dnf install -y pacemaker pcs corosync fence-agents-all --nogpgcheck

Open the cluster service ports through the firewall.
console
```
$ sudo firewall-cmd --permanent --add-service=high-availability
```
Reload the firewall.
console
```
$ sudo firewall-cmd --reload
```
Enable and start the pcs daemon.
console
```
$ sudo systemctl enable --now pcsd
```
Set the password for the hacluster user. Replace CLUSTER-PASSWORD with a strong password.
console
```
$ echo "hacluster:CLUSTER-PASSWORD" | sudo chpasswd
```

Important

Corosync requires each node's hostname to resolve to a single IPv4 address that is identical on both nodes and points to the cluster network, not a loopback address. Cloud images that use cloud-init map the hostname to both 127.0.0.1 and ::1 in /etc/hosts, which causes Corosync to fail with a "different IP families" error. On both nodes, set manage_etc_hosts: false in /etc/cloud/cloud.cfg, remove the loopback entries for the node hostnames from /etc/hosts, and add one IPv4 entry per node that maps each hostname to its VPC address (for example, 10.42.0.4 lustre-mds and 10.42.0.7 lustre-mds2).

Form the Cluster

Run the following steps from the primary MDS, which coordinates the configuration on both nodes.

Authenticate both nodes against pcsd. Replace the hostnames with your MDS hostnames and CLUSTER-PASSWORD with the password you set.
console
```
$ sudo pcs host auth MDS-PRIMARY-HOSTNAME MDS-SECONDARY-HOSTNAME -u hacluster -p CLUSTER-PASSWORD
```

Create the cluster.

                            console
                            
$ sudo pcs cluster setup lustre-cluster MDS-PRIMARY-HOSTNAME MDS-SECONDARY-HOSTNAME --force

Start the cluster on all nodes.
console
```
$ sudo pcs cluster start --all
```
Enable the cluster to start automatically on boot.
console
```
$ sudo pcs cluster enable --all
```
Verify the cluster status.
console
```
$ sudo pcs status
```
The output reports both nodes as Online and the partition as having quorum.

Define the MDT Resource

Run the following step on the primary MDS. The cluster manages the MDT mount as a Pacemaker resource that fails over to the standby node when the active node becomes unavailable.

Define the file system resource on the shared block device. Replace SHARED-DEVICE with the path to your shared block device.

                            console
                            
$ sudo pcs resource create mdt-resource ocf:heartbeat:Filesystem device=/dev/SHARED-DEVICE directory=/mnt/mdt fstype=lustre op monitor interval=30s

Note

Production deployments require fencing devices (STONITH) that match the underlying infrastructure. Configure STONITH agents specific to your platform before placing the cluster into production. The default stonith-enabled=false setting suits initial verification only.

Verify the Deployment

Benchmarks confirm that the deployment delivers the expected throughput. IOR measures parallel I/O throughput, and mdtest measures metadata operation throughput. Both tools run from the client node.

Build IOR and mdtest

IOR ships its source code on GitHub, which the Lustre client compiles from source. The mdtest binary builds alongside IOR.

Install the build dependencies.

                            console
                            
$ sudo dnf install -y gcc gcc-c++ make openmpi openmpi-devel git autoconf automake libtool --nogpgcheck

Load the OpenMPI environment module.
console
```
$ source /etc/profile.d/modules.sh
```
Load the OpenMPI module.
console
```
$ module load mpi/openmpi-x86_64
```

Clone the IOR repository.

                            console
                            
$ git clone https://github.com/hpc/ior.git ~/ior

Enter the IOR directory.
console
```
$ cd ~/ior
```
Check out the 4.0.0 release tag for a reproducible build.
console
```
$ git checkout 4.0.0
```
Generate the build configuration.
console
```
$ ./bootstrap
```
Run the configure script.
console
```
$ ./configure
```
Compile IOR and mdtest.
console
```
$ make
```

Run the Benchmarks

The benchmarks read and write inside dedicated directories on the Lustre file system.

Create a benchmark directory.
console
```
$ sudo mkdir -p /mnt/lustre/iortest-dir
```
Set permissions so the benchmark can write to the directory.
console
```
$ sudo chmod 777 /mnt/lustre/iortest-dir
```
Navigate to the IOR source directory.
console
```
$ cd ~/ior/src
```
Run IOR with a 100 MiB block size and a 1 MiB transfer size.
console
```
$ ./ior -w -r -t 1M -b 100M -F -o /mnt/lustre/iortest-dir/testfile
```
- -w -r: Performs both write and read tests.
- -t 1M: Sets the transfer size to 1 MiB.
- -b 100M: Sets the block size per process to 100 MiB.
- -F: Uses file-per-process mode.
The output reports the maximum write and read throughput in MiB/s.
Create the mdtest directory.
console
```
$ sudo mkdir -p /mnt/lustre/mdtest-dir
```
Set permissions so mdtest can write to the directory.
console
```
$ sudo chmod 777 /mnt/lustre/mdtest-dir
```
Run mdtest with 100 files per iteration and 2 iterations.
console
```
$ ./mdtest -n 100 -i 2 -d /mnt/lustre/mdtest-dir
```
The output reports rates for directory creation, file creation, file stat, and file removal.

Note

The benchmarks may print PSM3 warnings such as Failed to get eth0 (unit 0) cpu set on cloud Ethernet networks. The warnings are harmless and indicate that OpenMPI fell back from InfiniBand-style transport to standard Ethernet. The benchmark results remain valid.

Migrate from AWS FSx for Lustre to Self-Hosted Lustre

This section outlines the strategy for migrating workloads from AWS FSx for Lustre to a self-hosted Lustre deployment. It focuses on the concepts and the tools involved so the approach stays applicable as environments and tooling versions change. Consult the current AWS FSx for Lustre documentation and the Lustre manual for exact commands and options when you plan your migration.

Data Migration

Data migration moves the existing data set from AWS FSx to self-hosted Lustre through an intermediate S3 bucket, using stable, general-purpose interfaces rather than version-specific tooling. AWS FSx for Lustre links to an S3 bucket through a Data Repository Association, so the source side exports the file system contents to that bucket with an FSx data repository export task (aws fsx create-data-repository-task --type EXPORT_TO_REPOSITORY). On the destination side, a client that has the self-hosted Lustre file system mounted pulls the data down from the bucket with the AWS CLI (aws s3 sync) and writes it into a directory on the Lustre mount.

Because aws s3 sync accepts an --endpoint-url, the same approach works against any S3-compatible endpoint, which keeps the migration portable across AWS S3, Vultr Object Storage, and on-premises object stores. For large data sets, parallelize the transfer by running the sync from multiple clients against different prefixes, or copy directly between two mounted file systems with lfs migrate and rsync.

Client Migration

Client migration redirects each compute client from the FSx DNS endpoint to the self-hosted MDS address. Because Lustre presents the same POSIX interface on both sides, applications continue to work without code changes. On each client, unmount the FSx file system and mount the self-hosted file system in its place with mount -t lustre MDS-IP-ADDRESS@tcp:/lustrefs, using the MDS address on the Lustre network and your file system name. After remounting, applications read and write through the same paths they used with FSx.

S3 Integration Migration

Lustre Hierarchical Storage Management (HSM) is the self-hosted equivalent of the FSx S3 Data Repository. It tiers data automatically between Lustre and an S3-compatible object storage backend such as Vultr Object Storage. The HSM coordinator runs on the MDS (enabled through the mdt.*.hsm_control parameter), a copytool such as lhsmtool_posix runs on a client, and the backend is any POSIX directory that an S3 bucket exposes through a FUSE mount like s3fs. With HSM active, lfs hsm_archive copies a file to the S3 backend, lfs hsm_release frees the local copy while keeping the file in the namespace, and the next read transparently restores the content from S3. A policy engine such as Robinhood automates archive and release decisions based on age, size, or capacity thresholds.

Note

With an s3fs backend, the copytool logs fsetxattr ... Operation not supported messages during archive because s3fs does not implement extended attributes. The archive and restore operations still succeed, and the file's stripe layout is stored in a companion object alongside the data.

Data Storage Considerations

FSx for Lustre manages capacity expansion automatically, while a self-hosted deployment requires manual OST expansion. Plan capacity for the MDT and the OSTs based on expected workload patterns.

MDT sizing: Each file consumes approximately 2 KiB of MDT space, so plan for at least 1 GiB of MDT capacity per million files.
OST sizing: Calculate the aggregate capacity across all OSTs, then add 20 percent overhead for file system metadata and a growth buffer.
Adding capacity: Format new OST volumes with the next sequential index and mount them on a new or existing OSS node. The MGS recognizes new OSTs automatically, and the MDS distributes new file stripes across the expanded set.

Things to Take Care During Migration

The following items require attention during the migration window:

Auto-scaling differences: FSx scales storage transparently, while self-hosted Lustre requires manual OST addition. Provision sufficient initial OST capacity to cover projected growth.
S3 sync direction: FSx supports bidirectional automatic sync between the file system and S3. Self-hosted HSM uses explicit archive and restore operations, so writes to Lustre do not propagate to S3 automatically without a policy engine such as Robinhood.
Kernel module compatibility: Lustre client kernel modules must match the kernel version on each client. Update client packages whenever the client kernel changes, and pin the kernel on server nodes.
LNET configuration: The default TCP transport works for most cloud environments. InfiniBand deployments require additional LNET configuration on every node, with a matching transport on both servers and clients.
Metrics migration: FSx integrates with CloudWatch for metrics and alarming. The self-hosted equivalent uses the Lustre Prometheus exporter and Grafana dashboards, which require additional setup but provide deeper customization.
Backup strategy: FSx provides automated snapshots, while self-hosted Lustre requires explicit backup procedures. Combine lfs find with tar or rsync to back up file data, and use MDT snapshots through LDISKFS or LVM at the storage layer for the metadata.
Cost model: FSx applies per-TiB pricing that scales with capacity, while self-hosted Lustre cost depends on the underlying compute and storage. Calculate the breakeven capacity that justifies the operational overhead of self-management.

Conclusion

You have deployed a Lustre file system cluster as an alternative to AWS FSx for Lustre, with a Metadata Server, two Object Storage Servers, and a client, then configured striping and Progressive File Layouts, LNET networking, quotas and POSIX ACLs, Prometheus and Grafana monitoring, Pacemaker high availability, and HSM-based S3 integration. The cluster delivers parallel I/O throughput across multiple OSTs and supports the full Lustre POSIX interface that applications expect from FSx. For advanced tuning options such as Distributed Namespace (DNE), network striping, and the Robinhood policy engine, see the official Lustre documentation.

Tags:

Open Source

AWS

How to Deploy Lustre as an AWS FSx Alternative

How to Deploy Lustre as an AWS FSx Alternative

Understanding Lustre Architecture

Prerequisites

Install Lustre Packages

Add the Lustre Repository

Install Server Packages

Install Client Packages

Open the Lustre Network Port

Deploy the Metadata Server

Deploy Object Storage Servers

Configure the First OSS

Configure the Second OSS

Mount the Filesystem on Clients

Configure Striping and File Layouts

Set a Default Stripe Layout

Configure Progressive File Layouts

Set Up Networking with LNET

Configure Quotas and Access Control

Enable Quotas

Apply Quota Limits

Configure POSIX ACLs

Set Up Monitoring

Enable Jobstats

Build the Lustre Exporter

Install the Exporter as a Service

Deploy Prometheus

Install Grafana

Configure High Availability

Install Cluster Packages

Form the Cluster

Define the MDT Resource

Verify the Deployment

Build IOR and mdtest

Run the Benchmarks

Migrate from AWS FSx for Lustre to Self-Hosted Lustre

Data Migration

Client Migration

S3 Integration Migration

Data Storage Considerations

Things to Take Care During Migration

Conclusion

Comments

Tech Talks

Vultr Blogs