How to Deploy CephFS as a GCP Filestore Alternative

Google Cloud Filestore is Google Cloud's managed NFS file storage service that provides shared file systems for Compute Engine instances and Google Kubernetes Engine workloads. It removes the need to operate file servers, but it bills for provisioned capacity instead of consumed capacity, enforces per-tier minimum instance sizes and fixed performance tiers, and confines each instance to a single Google Cloud zone or region with access governed by Google Cloud IAM and VPC peering. Teams that need shared storage on their own infrastructure, with control over replication, performance tuning, and capacity growth, require a self-managed alternative.

CephFS is a POSIX-compliant distributed file system built on the Ceph storage platform that provides the same ReadWriteMany shared storage semantics as Filestore without the managed-service constraints. It provides shared storage across multiple nodes and can be deployed on Kubernetes using the Rook operator. CephFS supports dynamic volume provisioning through the Container Storage Interface (CSI) and can be extended using NFS-Ganesha to provide NFS access for external clients.

This article explains how to deploy CephFS on a Kubernetes Engine cluster using Rook. It covers installing the Rook operator, creating a Ceph cluster, configuring a CephFS filesystem, provisioning persistent volumes, and exporting storage over NFS. It also covers verifying storage functionality, configuring snapshots and monitoring, and preparing the environment for workload integration.

Understanding CephFS Architecture

CephFS maps directly to Google Cloud Filestore features while providing a self-managed and Kubernetes-native storage platform.

Key components of the CephFS deployment include:

• Rook Operator: Deploys and manages Ceph services inside the Kubernetes cluster.
• Ceph Monitor (MON) and Manager (MGR) Services: Maintain cluster health, quorum, and monitoring services.
• Object Storage Daemons (OSDs): Store Ceph data on attached block storage devices.
• Metadata Server (MDS) Pods: Manage CephFS metadata and coordinate filesystem access.
• Ceph CSI Driver: Dynamically provisions CephFS storage volumes for Kubernetes workloads.
• NFS-Ganesha: Exports CephFS storage over NFS for external clients.
• Prometheus and Grafana: Monitor Ceph cluster health, storage usage, and filesystem activity.

Prerequisites

Before you begin, you need to:

• Deploy a Kubernetes Engine cluster with at least 3 worker nodes, each attached to a raw unformatted Block Storage volume.
• Install kubectl and configure access to the Kubernetes cluster.
• Install Helm 3 on your management workstation.
• Have basic knowledge of Kubernetes storage concepts.

Install the Rook Ceph Operator

Rook extends Kubernetes with orchestration capabilities for Ceph storage services such as MONs, MGRs, OSDs, and MDSs.

1. Create the project directory.

   CONSOLE

   Copy

   $ mkdir -p ~/rook-ceph

   Explain Code

   Powered by Vultr AgentBeta

2. Switch to the project directory.

   CONSOLE

   Copy

   $ cd ~/rook-ceph

   Explain Code

   Powered by Vultr AgentBeta

3. Verify that all Kubernetes nodes are in the Ready state.

   CONSOLE

   Copy

   $ kubectl get nodes

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                         STATUS   ROLES    AGE   VERSION
   cephfs-worker-0166bfafa883   Ready    <none>   13m   v1.35.2
   cephfs-worker-375d99e1c630   Ready    <none>   14m   v1.35.2
   cephfs-worker-3a9f3e64fc10   Ready    <none>   14m   v1.35.2

4. Add the Rook Helm repository.

   CONSOLE

   Copy

   $ helm repo add rook-release https://charts.rook.io/release

   Explain Code

   Powered by Vultr AgentBeta

5. Update the local Helm repository index.

   CONSOLE

   Copy

   $ helm repo update

   Explain Code

   Powered by Vultr AgentBeta

6. Install the Rook Ceph operator using the Rook Ceph Helm chart.

   CONSOLE

   Copy

   $ helm install rook-ceph rook-release/rook-ceph --namespace rook-ceph --create-namespace --version v1.19.7

   Explain Code

   Powered by Vultr AgentBeta

   The command installs the Rook operator and the required Kubernetes custom resource definitions (CRDs).

   Note

   This article pins the Rook operator to v1.19.7, a tested release that automatically provisions the Ceph CSI driver used for dynamic CephFS volume provisioning. If you install a newer Rook version, confirm that the rook-ceph.cephfs.csi.ceph.com CSI driver is created before provisioning PersistentVolumeClaims.

7. Verify that the Rook Ceph operator resources are deploying.

   CONSOLE

   Copy

   $ kubectl get pods -n rook-ceph

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                                          READY   STATUS    RESTARTS   AGE
   ceph-csi-controller-manager-8bcc6877c-mwtxx   1/1     Running   0          11m
   rook-ceph-operator-64bb557454-kdjbt           1/1     Running   0          11m

8. Verify that the Ceph custom resource definitions (CRDs) are available before deploying the Ceph cluster.

   CONSOLE

   Copy

   $ kubectl get crds | grep ceph.rook.io

   Explain Code

   Powered by Vultr AgentBeta

Deploy the Ceph Cluster

The Ceph cluster provides the distributed storage backend used by CephFS. Rook deploys Ceph components as Kubernetes pods and manages storage devices attached to the worker nodes. The cluster is configured through the CephCluster custom resource.

1. Create the Ceph cluster configuration file.

   CONSOLE

   Copy

   $ nano ceph-cluster.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: ceph.rook.io/v1
   kind: CephCluster
   metadata:
     name: rook-ceph
     namespace: rook-ceph
   spec:
     cephVersion:
       image: quay.io/ceph/ceph:v19.2.3
     dataDirHostPath: /var/lib/rook
     mon:
       count: 3
       allowMultiplePerNode: false
     mgr:
       count: 1
       allowMultiplePerNode: false
     dashboard:
       enabled: true
       ssl: false
     storage:
       useAllNodes: true
       useAllDevices: false
       deviceFilter: "^vd[b-z]"
     resources:
       mgr:
         requests:
           cpu: "100m"
           memory: "1Gi"
         limits:
           memory: "2Gi"
       mon:
         requests:
           cpu: "250m"
           memory: "512Mi"
         limits:
           memory: "1Gi"
     healthCheck:
       daemonHealth:
         mon:
           interval: 45s
         osd:
           interval: 60s

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   Note

   The deviceFilter: "^vd[b-z]" setting instructs Rook to use only additional VirtIO block storage devices such as vdb, vdc, and vdd when creating Ceph OSDs. This excludes the primary system disk, which is typically attached as vda on cloud-based Kubernetes worker nodes. Verify the available block devices on each node before deployment and update the filter if your storage devices use a different naming scheme.

   The configuration:

   • deploys three Ceph monitor (MON) pods for cluster quorum
   • deploys one Ceph manager (MGR) pod
   • enables the Ceph dashboard
   • stores Ceph data in /var/lib/rook
   • provisions OSDs using attached block storage devices that match the ^vd[b-z] device filter
   • configures resource requests and limits for Ceph monitor and manager services

2. Deploy the Ceph cluster.

   CONSOLE

   Copy

   $ kubectl apply -f ceph-cluster.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Monitor the Ceph pods during deployment and wait for the monitor (MON), manager (MGR), and OSD pods to initialize.

   CONSOLE

   Copy

   $ kubectl get pods -n rook-ceph -w

   Explain Code

   Powered by Vultr AgentBeta

   Note

   The deployment process may take several minutes while Rook initializes the Ceph monitor (MON), manager (MGR), and object storage daemon (OSD) services. If the pods remain in a pending or failed state for an extended period, refer to the Rook Ceph Common Issues Documentation.

4. Verify that the OSD pods are running.

   CONSOLE

   Copy

   $ kubectl get pods -n rook-ceph | grep osd

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   rook-ceph-osd-0-68c9846b4-9862n                                   1/1     Running     0          5m40s
   rook-ceph-osd-1-5595c4798d-dwcmr                                  1/1     Running     0          5m40s
   rook-ceph-osd-2-689c4f4566-h9cxw                                  1/1     Running     0          5m40s
   rook-ceph-osd-prepare-ceph-worker-pool-2fd9cd11d442-89kzl         0/1     Completed   0          5m53s
   rook-ceph-osd-prepare-ceph-worker-pool-683ee0e17bef-nz26w         0/1     Completed   0          5m52s
   rook-ceph-osd-prepare-ceph-worker-pool-aac2507a4c09-8sgdr         0/1     Completed   0          5m51s

5. Verify that the Ceph cluster reaches the Ready state.

   CONSOLE

   Copy

   $ kubectl get cephcluster -n rook-ceph

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME        DATADIRHOSTPATH   MONCOUNT   AGE     PHASE   MESSAGE                        HEALTH      EXTERNAL   FSID
   rook-ceph   /var/lib/rook     3          8m52s   Ready   Cluster created successfully   HEALTH_OK              c9a844ce-fec1-4dbb-8f07-15144bfd8e1b

Create a CephFS Filesystem

The CephFS filesystem provides shared file storage backed by the Ceph cluster. Rook deploys Ceph metadata server (MDS) pods to manage filesystem metadata and coordinate file access between Kubernetes workloads. The filesystem is defined through the CephFilesystem custom resource.

1. Create the Ceph filesystem configuration file.

   CONSOLE

   Copy

   $ nano ceph-filesystem.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: ceph.rook.io/v1
   kind: CephFilesystem
   metadata:
     name: cephfs
     namespace: rook-ceph
   spec:
     metadataPool:
       replicated:
         size: 3
     dataPools:
       - replicated:
           size: 3
     preserveFilesystemOnDelete: true
     metadataServer:
       activeCount: 1
       activeStandby: true
       resources:
         requests:
           cpu: "100m"
           memory: "256Mi"
         limits:
           memory: "512Mi"

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration:

   • creates a CephFS filesystem named cephfs
   • provisions replicated metadata and data pools with three replicas
   • preserves the filesystem data if the Kubernetes resource is deleted
   • deploys one active metadata server (MDS) with one standby instance for failover
   • configures resource requests and limits for the MDS pods

2. Deploy the CephFS filesystem.

   CONSOLE

   Copy

   $ kubectl apply -f ceph-filesystem.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Monitor the CephFS resources during deployment and wait for the MDS pods to initialize.

   CONSOLE

   Copy

   $ kubectl get pods -n rook-ceph -w

   Explain Code

   Powered by Vultr AgentBeta

4. Verify that the MDS pods are running.

   CONSOLE

   Copy

   $ kubectl get pods -n rook-ceph | grep mds

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   rook-ceph-mds-cephfs-a-77f867b6b8-zzl64                           1/1     Running     0               104s
   rook-ceph-mds-cephfs-b-58d447d74f-ft9q5                           1/1     Running     0               103s

5. Verify that the Ceph filesystem reaches the Ready state.

   CONSOLE

   Copy

   $ kubectl get cephfilesystem -n rook-ceph

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME     ACTIVEMDS   AGE     PHASE
   cephfs   1           2m14s   Ready

Provision CephFS Volumes for Kubernetes

The CephFS CSI driver enables Kubernetes workloads to dynamically provision and mount shared CephFS storage volumes using PersistentVolumeClaims (PVCs).

1. Create the CephFS StorageClass configuration file.

   CONSOLE

   Copy

   $ nano cephfs-storageclass.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: cephfs-storage
   provisioner: rook-ceph.cephfs.csi.ceph.com
   parameters:
     clusterID: rook-ceph
     fsName: cephfs
     pool: cephfs-data0
     csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
     csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
     csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
     csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
     csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
     csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
   reclaimPolicy: Delete
   allowVolumeExpansion: true

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration:

   • creates a Kubernetes StorageClass named cephfs-storage
   • uses the Rook CephFS CSI driver for dynamic volume provisioning
   • provisions storage from the cephfs filesystem
   • enables dynamic volume expansion
   • automatically deletes persistent volumes when the associated PVC is removed

2. Deploy the StorageClass.

   CONSOLE

   Copy

   $ kubectl apply -f cephfs-storageclass.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Verify that the StorageClass is created.

   CONSOLE

   Copy

   $ kubectl get storageclass

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                             PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
   cephfs-storage                   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   49s
   vultr-block-storage (default)    block.csi.vultr.com             Delete          Immediate           true                   4h47m
   vultr-block-storage-hdd          block.csi.vultr.com             Delete          Immediate           true                   4h47m
   vultr-block-storage-hdd-retain   block.csi.vultr.com             Retain          Immediate           true                   4h47m
   vultr-block-storage-retain       block.csi.vultr.com             Retain          Immediate           true                   4h47m
   vultr-vfs-storage                block.csi.vultr.com             Delete          Immediate           true                   4h47m
   vultr-vfs-storage-retain         block.csi.vultr.com             Retain          Immediate           true                   4h47m

4. Create the PersistentVolumeClaim configuration file.

   CONSOLE

   Copy

   $ nano cephfs-pvc.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: cephfs-pvc
   spec:
     accessModes:
       - ReadWriteMany
     storageClassName: cephfs-storage
     resources:
       requests:
         storage: 5Gi

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration:

   • creates a PersistentVolumeClaim named cephfs-pvc
   • provisions a 5Gi CephFS-backed storage volume
   • enables multi-node read and write access using the ReadWriteMany access mode

5. Deploy the PersistentVolumeClaim.

   CONSOLE

   Copy

   $ kubectl apply -f cephfs-pvc.yaml

   Explain Code

   Powered by Vultr AgentBeta

6. Verify that the PersistentVolumeClaim is bound.

   CONSOLE

   Copy

   $ kubectl get pvc

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
   cephfs-pvc   Bound    pvc-bb4b78dd-140f-4b77-90eb-37c60a5134a8   5Gi        RWX            cephfs-storage   <unset>                 1h

7. Create the test pod configuration file.

   CONSOLE

   Copy

   $ nano cephfs-test-pod.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: v1
   kind: Pod
   metadata:
     name: cephfs-test-pod
   spec:
     containers:
       - name: cephfs-test-container
         image: busybox
         command: ["/bin/sh", "-c"]
         args:
           - while true; do sleep 30; done
         volumeMounts:
           - name: cephfs-storage
             mountPath: /mnt/cephfs
     volumes:
       - name: cephfs-storage
         persistentVolumeClaim:
           claimName: cephfs-pvc

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration mounts the CephFS-backed PersistentVolumeClaim inside the pod at /mnt/cephfs.

8. Deploy the test pod.

   CONSOLE

   Copy

   $ kubectl apply -f cephfs-test-pod.yaml

   Explain Code

   Powered by Vultr AgentBeta

9. Verify that the test pod is running.

   CONSOLE

   Copy

   $ kubectl get pod cephfs-test-pod

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME              READY   STATUS    RESTARTS   AGE
   cephfs-test-pod   1/1     Running   0          53s

10. Create a test file inside the mounted CephFS volume.

    CONSOLE

    Copy

    $ kubectl exec -it cephfs-test-pod -- sh

    Explain Code

    Powered by Vultr AgentBeta

    Inside the container, create a test file.

    CONSOLE

    Copy

    $ echo "CephFS volume test successful" > /mnt/cephfs/test.txt

    Explain Code

    Powered by Vultr AgentBeta

11. Verify that the file exists.

    CONSOLE

    Copy

    $ kubectl exec -it cephfs-test-pod -- cat /mnt/cephfs/test.txt

    Explain Code

    Powered by Vultr AgentBeta

    Output:

    CephFS volume test successful

Export CephFS via NFS

The Ceph NFS service enables external systems and legacy applications to access CephFS storage using the standard NFS protocol. Rook deploys NFS-Ganesha pods to export CephFS shares over the network. The NFS service is defined through the CephNFS custom resource, and exports are managed with the ceph nfs export commands.

1. Create the Ceph NFS configuration file.

   CONSOLE

   Copy

   $ nano ceph-nfs.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: ceph.rook.io/v1
   kind: CephNFS
   metadata:
     name: ceph-nfs
     namespace: rook-ceph
   spec:
     rados:
       pool: .nfs
     server:
       active: 1
       resources:
         requests:
           cpu: "100m"
           memory: "256Mi"
         limits:
           memory: "512Mi"

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration:

   • deploys an NFS-Ganesha server named ceph-nfs
   • creates a dedicated .nfs Reliable Autonomic Distributed Object Store (RADOS) pool for NFS recovery data
   • deploys one active NFS server pod
   • configures resource requests and limits for the NFS service

2. Deploy the NFS service.

   CONSOLE

   Copy

   $ kubectl apply -f ceph-nfs.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Monitor the NFS resources during deployment.

   CONSOLE

   Copy

   $ kubectl get pods -n rook-ceph -w

   Explain Code

   Powered by Vultr AgentBeta

4. Verify that the NFS-Ganesha pod is running.

   CONSOLE

   Copy

   $ kubectl get pods -n rook-ceph | grep nfs

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   rook-ceph-nfs-ceph-nfs-a-6bdf6c79c6-c47lr                         2/2     Running            0               48s

5. Create a Rook Ceph toolbox pod to manage Ceph NFS exports.

   CONSOLE

   Copy

   $ kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/toolbox.yaml

   Explain Code

   Powered by Vultr AgentBeta

6. Verify that the toolbox pod is running.

   CONSOLE

   Copy

   $ kubectl get pods -n rook-ceph | grep tools

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   rook-ceph-tools-7c66d65f8b-lh6mt                                  1/1     Running     0                 100s

7. Create an NFS export for the CephFS filesystem.

   CONSOLE

   Copy

   $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export create cephfs ceph-nfs /cephfs cephfs --path=/

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   {
     "bind": "/cephfs",
     "cluster": "ceph-nfs",
     "fs": "cephfs",
     "mode": "RW",
     "path": "/"
   }

   The command exports the cephfs filesystem through the ceph-nfs NFS-Ganesha service using the /cephfs export path.

8. Verify that the NFS export is created.

   CONSOLE

   Copy

   $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export ls ceph-nfs

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   [
     "/cephfs"
   ]

9. Retrieve the NFS service details.

   CONSOLE

   Copy

   $ kubectl get svc -n rook-ceph | grep nfs

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   rook-ceph-nfs-ceph-nfs-a   ClusterIP   10.102.91.113    <none>        2049/TCP,9587/TCP   34m

Mount the NFS Export from a Linux Client

The NFS service is created with a ClusterIP address, which is reachable from the Kubernetes worker nodes and from pods running in the cluster. Mount the exported NFS share from one of the worker nodes.

1. Create a temporary mount directory.

   CONSOLE

   Copy

   $ sudo mkdir -p /mnt/cephfs-nfs

   Explain Code

   Powered by Vultr AgentBeta

2. Install the NFS client utilities.

   • On Ubuntu and Debian:

     CONSOLE

     Copy

     $ sudo apt install nfs-common -y

     Explain Code

     Powered by Vultr AgentBeta

   • On Rocky Linux and AlmaLinux:

     CONSOLE

     Copy

     $ sudo dnf install nfs-utils -y

     Explain Code

     Powered by Vultr AgentBeta

3. Mount the exported CephFS NFS share. Replace NFS-SERVICE-IP with the ClusterIP address of the rook-ceph-nfs-ceph-nfs-a service.

   CONSOLE

   Copy

   $ sudo mount -t nfs NFS-SERVICE-IP:/cephfs /mnt/cephfs-nfs

   Explain Code

   Powered by Vultr AgentBeta

4. Create a test file in the mounted NFS share.

   CONSOLE

   Copy

   $ echo "NFS export test successful" | sudo tee /mnt/cephfs-nfs/test.txt

   Explain Code

   Powered by Vultr AgentBeta

5. Verify that the file exists.

   CONSOLE

   Copy

   $ cat /mnt/cephfs-nfs/test.txt

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NFS export test successful

   This confirms that the CephFS filesystem is exported through NFS-Ganesha and accessible using the standard NFS protocol.

Configure Storage Pools and Performance

Ceph storage pools control how data is distributed and replicated across the cluster. In addition to the pools that back the CephFS filesystem deployed earlier, you can create dedicated block pools for other Kubernetes workloads and configure replica placement policies to improve resiliency.

1. Create the Ceph block pool configuration file.

   CONSOLE

   Copy

   $ nano ceph-blockpool.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: ceph.rook.io/v1
   kind: CephBlockPool
   metadata:
     name: app-block-pool
     namespace: rook-ceph
   spec:
     failureDomain: host
     replicated:
       size: 3
     parameters:
       compression_mode: none

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration:

   • creates a replicated Ceph block pool named app-block-pool, independent of the pools backing the CephFS filesystem, for workloads that need RBD-backed block storage
   • stores three copies of each object across different Kubernetes worker nodes
   • uses the host failure domain to distribute replicas across separate nodes
   • disables compression for predictable storage performance

2. Deploy the storage pool.

   CONSOLE

   Copy

   $ kubectl apply -f ceph-blockpool.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Verify that the storage pool is created.

   CONSOLE

   Copy

   $ kubectl get cephblockpool -n rook-ceph

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME              PHASE   TYPE         FAILUREDOMAIN   AGE
   app-block-pool    Ready   Replicated   host            11s

4. View the Ceph cluster health.

   CONSOLE

   Copy

   $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   cluster:
       id:     c9a844ce-fec1-4dbb-8f07-15144bfd8e1b
       health: HEALTH_OK
   
     services:
       mon: 3 daemons, quorum a,b,c (age 21h)
       mgr: a(active, since 9s)
       mds: 1/1 daemons up, 1 hot standby
       osd: 3 osds: 3 up (since 21h), 3 in (since 21h)
   
     data:
       volumes: 1/1 healthy
       pools:   5 pools, 113 pgs
       objects: 35 objects, 1.3 MiB
       usage:   248 MiB used, 304 GiB / 304 GiB avail
       pgs:     113 active+clean
   
     io:
       client:   26 KiB/s rd, 0 B/s wr, 10 op/s rd, 1 op/s wr

   This confirms that the Ceph block pool is configured and the cluster is operating in a healthy state.

Set Up Snapshots and Backups

CephFS snapshots allow you to capture point-in-time copies of persistent volumes and restore data when needed. Kubernetes uses the CSI snapshot controller together with the Rook CephFS CSI driver to manage volume snapshot operations.

Install the Kubernetes CSI Snapshot CRDs

The Kubernetes CSI snapshot CRDs enable Kubernetes to manage volume snapshots for CSI-compatible storage drivers such as Rook CephFS.

1. Deploy the VolumeSnapshotClass CRD.

   CONSOLE

   Copy

   $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml

   Explain Code

   Powered by Vultr AgentBeta

2. Deploy the VolumeSnapshotContent CRD.

   CONSOLE

   Copy

   $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Deploy the VolumeSnapshot CRD.

   CONSOLE

   Copy

   $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

   Explain Code

   Powered by Vultr AgentBeta

4. Deploy the snapshot controller RBAC resources.

   CONSOLE

   Copy

   $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml

   Explain Code

   Powered by Vultr AgentBeta

5. Deploy the snapshot controller.

   CONSOLE

   Copy

   $ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

   Explain Code

   Powered by Vultr AgentBeta

6. Verify that the snapshot controller pods are running.

   CONSOLE

   Copy

   $ kubectl get pods -n kube-system | grep snapshot

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   snapshot-controller-74c7dc46c-c7xbv        1/1     Running   0          33s
   snapshot-controller-74c7dc46c-lmdlv        1/1     Running   0          33s

Create a VolumeSnapshotClass

The VolumeSnapshotClass defines how Kubernetes creates and manages snapshots using the Rook CephFS CSI driver.

1. Create the snapshot class configuration file.

   CONSOLE

   Copy

   $ nano cephfs-snapshotclass.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshotClass
   metadata:
     name: cephfs-snapshotclass
   driver: rook-ceph.cephfs.csi.ceph.com
   deletionPolicy: Delete
   parameters:
     clusterID: rook-ceph
     csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
     csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration:

   • creates a snapshot class named cephfs-snapshotclass
   • uses the Rook CephFS CSI driver for snapshot operations
   • configures the CephFS snapshotter secret used by the CSI driver
   • automatically deletes snapshots when the snapshot resource is removed

2. Deploy the snapshot class.

   CONSOLE

   Copy

   $ kubectl apply -f cephfs-snapshotclass.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Verify that the snapshot class is created.

   CONSOLE

   Copy

   $ kubectl get volumesnapshotclass

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                   DRIVER                          DELETIONPOLICY   AGE
   cephfs-snapshotclass   rook-ceph.cephfs.csi.ceph.com   Delete           12s

Create a CephFS Volume Snapshot

The VolumeSnapshot resource creates a point-in-time snapshot of a PersistentVolumeClaim using the CephFS CSI driver.

1. Create the snapshot configuration file.

   CONSOLE

   Copy

   $ nano cephfs-snapshot.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: snapshot.storage.k8s.io/v1
   kind: VolumeSnapshot
   metadata:
     name: cephfs-snapshot
   spec:
     volumeSnapshotClassName: cephfs-snapshotclass
     source:
       persistentVolumeClaimName: cephfs-pvc

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration creates a snapshot of the cephfs-pvc PersistentVolumeClaim using the cephfs-snapshotclass snapshot class.

2. Deploy the snapshot.

   CONSOLE

   Copy

   $ kubectl apply -f cephfs-snapshot.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Verify that the snapshot is ready.

   CONSOLE

   Copy

   $ kubectl get volumesnapshot

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                  READYTOUSE   SOURCEPVC    SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS          SNAPSHOTCONTENT                                    CREATIONTIME   AGE
   cephfs-snapshot       true         cephfs-pvc                           5Gi           cephfs-snapshotclass   snapcontent-06f8cb1b-a46a-4de9-9aa8-eb7ddd34aa04   46s            47s

Restore a Snapshot to a New PersistentVolumeClaim

You can restore a CephFS snapshot to a new PersistentVolumeClaim to recover application data or create a duplicate volume from an existing snapshot.

1. Create the restore PVC configuration file.

   CONSOLE

   Copy

   $ nano cephfs-restore-pvc.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: cephfs-restore-pvc
   spec:
     accessModes:
       - ReadWriteMany
     storageClassName: cephfs-storage
     resources:
       requests:
         storage: 5Gi
     dataSource:
       name: cephfs-snapshot
       kind: VolumeSnapshot
       apiGroup: snapshot.storage.k8s.io

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration:

   • creates a new PersistentVolumeClaim named cephfs-restore-pvc
   • restores data from the cephfs-snapshot snapshot
   • uses the cephfs-storage StorageClass for CephFS storage
   • provisions a new 5Gi restored volume from the snapshot source

   Note

   The restored PersistentVolumeClaim size must be equal to or larger than the source snapshot restore size. Kubernetes does not allow restoring snapshots into smaller volumes.

2. Deploy the restored PersistentVolumeClaim.

   CONSOLE

   Copy

   $ kubectl apply -f cephfs-restore-pvc.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Verify that the restored PersistentVolumeClaim is bound.

   CONSOLE

   Copy

   $ kubectl get pvc

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
   cephfs-pvc           Bound    pvc-bb4b78dd-140f-4b77-90eb-37c60a5134a8   5Gi        RWX            cephfs-storage   <unset>                 40h
   cephfs-restore-pvc   Bound    pvc-338939cf-72a3-45b9-a978-31a54e29ead4   5Gi        RWX            cephfs-storage   <unset>                 31s

Configure Scheduled CephFS Snapshots

Kubernetes does not create scheduled snapshots automatically. You can use a Kubernetes CronJob resource to create recurring CephFS snapshots for backup and recovery workflows. The CronJob runs kubectl inside the cluster, so it requires a ServiceAccount with role-based access control (RBAC) permission to manage VolumeSnapshot resources.

1. Create the snapshot RBAC configuration file.

   CONSOLE

   Copy

   $ nano cephfs-snapshot-rbac.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: cephfs-snapshot-sa
     namespace: default
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: cephfs-snapshot-role
     namespace: default
   rules:
     - apiGroups: ["snapshot.storage.k8s.io"]
       resources: ["volumesnapshots"]
       verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: cephfs-snapshot-rolebinding
     namespace: default
   subjects:
     - kind: ServiceAccount
       name: cephfs-snapshot-sa
       namespace: default
   roleRef:
     kind: Role
     name: cephfs-snapshot-role
     apiGroup: rbac.authorization.k8s.io

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration:

   • creates a ServiceAccount named cephfs-snapshot-sa for the snapshot CronJob
   • grants a Role that allows managing VolumeSnapshot resources in the default namespace
   • binds the Role to the ServiceAccount

2. Deploy the snapshot RBAC resources.

   CONSOLE

   Copy

   $ kubectl apply -f cephfs-snapshot-rbac.yaml

   Explain Code

   Powered by Vultr AgentBeta

3. Create the scheduled snapshot configuration file.

   CONSOLE

   Copy

   $ nano cephfs-snapshot-cronjob.yaml

   Explain Code

   Powered by Vultr AgentBeta

   Add the following configuration to the file:

   YAML

   Copy

   apiVersion: batch/v1
   kind: CronJob
   metadata:
     name: cephfs-snapshot-cronjob
   spec:
     schedule: "0 */6 * * *"
     jobTemplate:
       spec:
         template:
           spec:
             serviceAccountName: cephfs-snapshot-sa
             restartPolicy: OnFailure
             containers:
               - name: snapshot-creator
                 image: alpine/k8s:1.36.2
                 command:
                   - /bin/sh
                   - -c
                   - |
                     SNAPSHOT_NAME="cephfs-snapshot-$(date +%s)"
                     cat <<EOF | kubectl apply -f -
                     apiVersion: snapshot.storage.k8s.io/v1
                     kind: VolumeSnapshot
                     metadata:
                       name: ${SNAPSHOT_NAME}
                     spec:
                       volumeSnapshotClassName: cephfs-snapshotclass
                       source:
                         persistentVolumeClaimName: cephfs-pvc
                     EOF

   Explain Code

   Powered by Vultr AgentBeta

   Save and close the file.

   The configuration:

   • creates a Kubernetes CronJob named cephfs-snapshot-cronjob
   • runs under the cephfs-snapshot-sa ServiceAccount with permission to manage snapshots
   • creates a new CephFS snapshot every 6 hours
   • generates unique snapshot names using timestamps
   • uses the cephfs-snapshotclass snapshot class
   • creates snapshots of the cephfs-pvc PersistentVolumeClaim

4. Deploy the scheduled snapshot CronJob.

   CONSOLE

   Copy

   $ kubectl apply -f cephfs-snapshot-cronjob.yaml

   Explain Code

   Powered by Vultr AgentBeta

5. Verify that the CronJob is created.

   CONSOLE

   Copy

   $ kubectl get cronjob

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                      SCHEDULE      TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
   cephfs-snapshot-cronjob   0 */6 * * *   <none>     False     0        <none>          10s

Configure Authentication and Access Control

CephFS uses CephX authentication to control client access to the filesystem. You can create restricted CephFS users, limit access to specific directories, and apply POSIX file permissions to isolate workloads between tenants and applications.

Configure POSIX File Permissions

CephFS honors standard POSIX ownership and permission bits, so you can isolate directories for different tenants using the same chown and chmod commands you would use on a local filesystem.

1. Open a shell inside the test pod.

   CONSOLE

   Copy

   $ kubectl exec -it cephfs-test-pod -- sh

   Explain Code

   Powered by Vultr AgentBeta

2. Create separate directories for different tenants inside the mounted CephFS volume.

   CONSOLE

   Copy

   $ mkdir -p /mnt/cephfs/tenant-a /mnt/cephfs/tenant-b

   Explain Code

   Powered by Vultr AgentBeta

3. Assign ownership to the tenant-a directory.

   CONSOLE

   Copy

   $ chown 1001:1001 /mnt/cephfs/tenant-a

   Explain Code

   Powered by Vultr AgentBeta

4. Restrict access to the directory owner only.

   CONSOLE

   Copy

   $ chmod 700 /mnt/cephfs/tenant-a

   Explain Code

   Powered by Vultr AgentBeta

5. Verify the directory permissions.

   CONSOLE

   Copy

   $ ls -ld /mnt/cephfs/tenant-a

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   drwx------    2 1001     1001             0 May 28 00:12 /mnt/cephfs/tenant-a

6. Exit the pod shell.

   CONSOLE

   Copy

   $ exit

   Explain Code

   Powered by Vultr AgentBeta

Create Restricted CephFS Client Users

CephFS supports path-based authorization using CephX authentication. You can create dedicated CephFS users and restrict access to specific filesystem paths for tenant isolation and workload security.

1. Open a shell inside the Rook toolbox pod.

   CONSOLE

   Copy

   $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- bash

   Explain Code

   Powered by Vultr AgentBeta

2. Create a CephFS client user with read and write access restricted to the /tenant-a directory.

   CONSOLE

   Copy

   $ ceph fs authorize cephfs client.tenant-a /tenant-a rw

   Explain Code

   Powered by Vultr AgentBeta

3. Retrieve the authentication key for the client.tenant-a CephFS user.

   CONSOLE

   Copy

   $ ceph auth get-key client.tenant-a

   Explain Code

   Powered by Vultr AgentBeta

4. View the client capabilities.

   CONSOLE

   Copy

   $ ceph auth get client.tenant-a

   Explain Code

   Powered by Vultr AgentBeta

5. Exit the toolbox shell.

   CONSOLE

   Copy

   $ exit

   Explain Code

   Powered by Vultr AgentBeta

   Note

   The path passed to ceph fs authorize is resolved from the root of the CephFS filesystem, not from the pod's local PersistentVolumeClaim mount point. If the CSI driver provisions the PersistentVolumeClaim as a subvolume rather than exposing the filesystem root directly, retrieve the subvolume's actual path with ceph fs subvolume getpath before authorizing access, and adjust the path argument to match.

6. Verify that the client.tenant-a credentials restrict access to the intended path. Mount CephFS directly on a Linux client using the kernel client and the retrieved key. Replace MON-IP with the IP address of a Ceph monitor, and CLIENT-KEY with the key retrieved for client.tenant-a in the previous step.

   CONSOLE

   Copy

   $ sudo mount -t ceph MON-IP:6789:/tenant-a /mnt/tenant-a-test -o name=tenant-a,secret=CLIENT-KEY

   Explain Code

   Powered by Vultr AgentBeta

7. Confirm that the mount succeeds and that only the authorized directory is visible.

   CONSOLE

   Copy

   $ ls /mnt/tenant-a-test

   Explain Code

   Powered by Vultr AgentBeta

   A permission error when accessing paths outside the mounted directory confirms that the CephX authorization correctly restricts the client's access.

Set Up Monitoring

Ceph provides built-in monitoring features through the Ceph Dashboard, Prometheus metrics endpoints, and Grafana dashboards. You can use these services to monitor cluster health, storage utilization, CephFS activity, and OSD performance.

Verify that the Ceph Dashboard Is Enabled

The Ceph Manager (MGR) service hosts the Ceph Dashboard and monitoring modules.

1. Verify that the Ceph cluster is healthy.

   CONSOLE

   Copy

   $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status

   Explain Code

   Powered by Vultr AgentBeta

   Output:

     cluster:
       id:     c9a844ce-fec1-4dbb-8f07-15144bfd8e1b
       health: HEALTH_OK
   
     services:
       mon: 3 daemons, quorum a,b,c (age 47h)
       mgr: a(active, since 3h)
       mds: 1/1 daemons up, 1 hot standby
       osd: 3 osds: 3 up (since 47h), 3 in (since 47h)
   
     data:
       volumes: 1/1 healthy
       pools:   5 pools, 113 pgs
       objects: 46 objects, 3.9 MiB
       usage:   502 MiB used, 304 GiB / 304 GiB avail
       pgs:     113 active+clean
   
     io:
       client:   938 B/s rd, 1 op/s rd, 0 op/s wr

2. Verify that the Ceph Manager pod is running.

   CONSOLE

   Copy

   $ kubectl get pods -n rook-ceph | grep mgr

   Explain Code

   Powered by Vultr AgentBeta

3. Retrieve the Ceph Dashboard service details.

   CONSOLE

   Copy

   $ kubectl get svc -n rook-ceph | grep dashboard

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   rook-ceph-mgr-dashboard    ClusterIP   10.111.11.76     <none>        7000/TCP            47h

Access the Ceph Dashboard

The Ceph Dashboard service defaults to a ClusterIP address, so reach it from your local workstation using a kubectl port-forward session.

1. Forward the Ceph Dashboard service to your local workstation.

   CONSOLE

   Copy

   $ kubectl port-forward -n rook-ceph svc/rook-ceph-mgr-dashboard 7000:7000

   Explain Code

   Powered by Vultr AgentBeta

2. Open http://127.0.0.1:7000 in a web browser.

3. Retrieve the dashboard administrator password. Open a new terminal session and run the following command.

   CONSOLE

   Copy

   $ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

   Explain Code

   Powered by Vultr AgentBeta

   Copy the password from the command output.

4. Use the default username admin and paste the copied password to log in to the Ceph Dashboard.

   

Enable the Prometheus Monitoring Module

Ceph exports monitoring metrics through the Prometheus module running on the Ceph Manager service.

1. Enable the Prometheus module.

   CONSOLE

   Copy

   $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph mgr module enable prometheus

   Explain Code

   Powered by Vultr AgentBeta

2. Verify that the Prometheus module is enabled.

   CONSOLE

   Copy

   $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph mgr module ls | grep prometheus

   Explain Code

   Powered by Vultr AgentBeta

3. Retrieve the Ceph Manager service details.

   CONSOLE

   Copy

   $ kubectl get svc -n rook-ceph | grep mgr

   Explain Code

   Powered by Vultr AgentBeta

Deploy Grafana Dashboards

Grafana visualizes Prometheus metrics using dashboards for Ceph cluster monitoring.

1. Add the Grafana Community Helm repository.

   CONSOLE

   Copy

   $ helm repo add grafana-community https://grafana-community.github.io/helm-charts

   Explain Code

   Powered by Vultr AgentBeta

2. Update the Helm repository index.

   CONSOLE

   Copy

   $ helm repo update

   Explain Code

   Powered by Vultr AgentBeta

3. Deploy Grafana.

   CONSOLE

   Copy

   $ helm install grafana grafana-community/grafana --namespace monitoring --create-namespace

   Explain Code

   Powered by Vultr AgentBeta

4. Verify that the Grafana pod is running.

   CONSOLE

   Copy

   $ kubectl get pods -n monitoring

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                       READY   STATUS    RESTARTS   AGE
   grafana-5786bf9b45-jml9z   1/1     Running   0          3m9s

5. Retrieve the Grafana administrator password.

   CONSOLE

   Copy

   $ kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

   Explain Code

   Powered by Vultr AgentBeta

   Copy the password value.

6. Forward the Grafana service to your local workstation.

   CONSOLE

   Copy

   $ kubectl port-forward -n monitoring svc/grafana 3000:80

   Explain Code

   Powered by Vultr AgentBeta

7. Open http://127.0.0.1:3000 in a web browser.

8. Use the default username admin and the password retrieved in Step 5 to log in to Grafana.

   

Verify the Deployment

You can perform an end-to-end validation to verify the CephFS deployment, NFS export, snapshot functionality, and monitoring services.

Verify CephFS PersistentVolumeClaim Access

Confirm that the PersistentVolumeClaim provisioned earlier remains bound and that the test pod can still read and write to the mounted CephFS volume.

1. Verify that the CephFS-backed PersistentVolumeClaim is bound.

   CONSOLE

   Copy

   $ kubectl get pvc

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
   cephfs-pvc           Bound    pvc-bb4b78dd-140f-4b77-90eb-37c60a5134a8   5Gi        RWX            cephfs-storage   <unset>                 44h
   cephfs-restore-pvc   Bound    pvc-338939cf-72a3-45b9-a978-31a54e29ead4   5Gi        RWX            cephfs-storage   <unset>                 4h32m

2. Verify that the CephFS test pod is running.

   CONSOLE

   Copy

   $ kubectl get pod cephfs-test-pod

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME              READY   STATUS    RESTARTS   AGE
   cephfs-test-pod   1/1     Running   0          31h

3. Create a validation file inside the mounted CephFS volume.

   CONSOLE

   Copy

   $ kubectl exec -it cephfs-test-pod -- sh -c 'echo "CephFS deployment verification successful" > /mnt/cephfs/verify.txt'

   Explain Code

   Powered by Vultr AgentBeta

4. Verify that the file exists.

   CONSOLE

   Copy

   $ kubectl exec -it cephfs-test-pod -- cat /mnt/cephfs/verify.txt

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   CephFS deployment verification successful

Verify the CephFS NFS Export

Confirm that the NFS-Ganesha export created earlier is still registered and that the underlying service remains reachable.

1. Verify that the NFS export is available through the Ceph NFS service.

   CONSOLE

   Copy

   $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export ls ceph-nfs

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   [
     "/cephfs"
   ]

2. Verify that the NFS service is running.

   CONSOLE

   Copy

   $ kubectl get svc -n rook-ceph | grep nfs

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   rook-ceph-nfs-ceph-nfs-a   ClusterIP   10.102.91.113    <none>        2049/TCP,9587/TCP   30h

Verify Snapshot and Restore Operations

Confirm that the snapshot and restore workflow completed successfully and that both the original and restored volumes remain available.

1. Verify that the CephFS snapshot is ready.

   CONSOLE

   Copy

   $ kubectl get volumesnapshot

   Explain Code

   Powered by Vultr AgentBeta

2. Verify that the restored PersistentVolumeClaim is bound.

   CONSOLE

   Copy

   $ kubectl get pvc cephfs-restore-pvc

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
   cephfs-restore-pvc   Bound    pvc-338939cf-72a3-45b9-a978-31a54e29ead4   5Gi        RWX            cephfs-storage   <unset>                 4h43m

Verify Monitoring Services

Confirm that the Ceph Dashboard and Grafana remain available and that the cluster continues reporting a healthy status after the storage operations performed throughout this article.

1. Verify that the Ceph Dashboard service is running.

   CONSOLE

   Copy

   $ kubectl get svc -n rook-ceph | grep dashboard

   Explain Code

   Powered by Vultr AgentBeta

2. Verify that the Grafana pod is running.

   CONSOLE

   Copy

   $ kubectl get pods -n monitoring

   Explain Code

   Powered by Vultr AgentBeta

   Output:

   NAME                       READY   STATUS    RESTARTS   AGE
   grafana-5786bf9b45-jml9z   1/1     Running   0          47m

3. Verify that the Ceph cluster remains healthy after all storage operations.

   CONSOLE

   Copy

   $ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status

   Explain Code

   Powered by Vultr AgentBeta

Migrate from GCP Filestore to CephFS

This section outlines the strategy for migrating workloads from GCP Filestore to CephFS on Kubernetes. It focuses on the concepts and the tools involved so the approach stays applicable as environments and tooling versions change. Refer to the current GCP Filestore and Rook/Ceph CephFS documentation for the exact commands and configuration options before planning your migration.

Data Migration

Data migration copies the existing data set from the Filestore NFS export to CephFS while preserving file permissions and ownership. Because both Filestore and the CephFS NFS-Ganesha export present a standard NFS interface, a client that mounts both file systems can transfer data between them with a general-purpose tool such as rsync. Mount the Filestore export and the CephFS export on the same client, then run rsync in archive mode (rsync -avh) so it preserves ownership, permissions, symbolic links, and timestamps during the copy. For large data sets, parallelize the transfer by running rsync from multiple clients against different subdirectories, or copy the data into a pod that has the CephFS PersistentVolumeClaim mounted.

Application Migration

Application migration redirects Kubernetes workloads from Filestore-backed storage to CephFS. Because CephFS provides the same POSIX and ReadWriteMany semantics that Filestore offers, most applications continue to work without code changes. Update each workload's PersistentVolumeClaim to reference the cephfs-storage StorageClass instead of the Filestore StorageClass (for example, filestore-rwx), then recreate the PersistentVolumeClaim and restart the workload so it binds to the new CephFS volume. For applications that run outside Kubernetes, repoint the NFS mount target at the CephFS NFS-Ganesha export instead of the Filestore endpoint, using the same mount -t nfs command shown earlier in this article.

Snapshot and Backup Migration

Kubernetes CSI VolumeSnapshot resources are the CephFS equivalent of Filestore snapshots. After the CSI snapshot controller and the cephfs-snapshotclass are in place, point-in-time snapshots of any CephFS PersistentVolumeClaim are created by referencing the claim from a VolumeSnapshot resource, as shown earlier in this article. If the existing environment uses scheduled Filestore backups, recreate the schedules with a Kubernetes CronJob or an external backup automation tool, and store long-term copies in an object storage backend.

Data Storage Considerations

GCP Filestore manages capacity expansion automatically, while CephFS requires manual OSD expansion and replication planning as storage usage grows. Plan capacity across the Ceph cluster based on expected workload patterns.

• Replication overhead: CephFS uses a replication factor of 3 by default, so raw storage usage is roughly three times the usable capacity. Account for this multiplier when sizing the OSD devices.
• Adding capacity: Attach additional Block Storage volumes to the worker nodes and let Rook provision new OSDs through the device filter. Ceph rebalances data across the expanded set automatically.
• Pool placement: Use separate Ceph pools or OSD device classes to place performance-sensitive workloads on faster storage, similar to Filestore capacity tiers.

Things to Consider During Migration

The following items require attention during the migration window:

• NFS protocol compatibility: Some Filestore deployments use NFSv3 clients. CephFS exports through NFS-Ganesha use NFSv4 by default. Verify client compatibility before migration.
• Storage performance tuning: Filestore performance tiers are managed automatically. CephFS performance depends on OSD device type, replication settings, network throughput, and Ceph pool configuration.
• Multi-share migration: Filestore multi-share environments map to CephFS subdirectories or separate CephFS subvolumes for tenant isolation.
• Authentication and access control: GCP Filestore integrates with Google Cloud Identity and Access Management (IAM) policies. CephFS uses CephX authentication together with POSIX permissions and ACLs.
• Monitoring migration: Replace Google Cloud Monitoring dashboards with the Ceph Dashboard, Prometheus metrics, and Grafana dashboards.
• Capacity planning: CephFS replication increases raw storage usage compared to single-copy storage systems. Plan storage growth before migrating production workloads.

Conclusion

You have deployed CephFS on a Kubernetes Engine cluster using the Rook operator and configured shared persistent storage for Kubernetes workloads. The setup includes dynamic CephFS volume provisioning, NFS exports using NFS-Ganesha, CSI snapshot and restore operations, CephX authentication, monitoring with the Ceph Dashboard and Grafana, and migration guidance from GCP Filestore. This configuration provides a scalable and self-managed distributed storage platform for containerized applications running on Kubernetes. For additional configuration options and production best practices, refer to the Rook documentation and Ceph documentation.