How to Deploy CephFS as a GCP Filestore Alternative

CephFS is a POSIX-compliant distributed file system built on the Ceph storage platform. It provides shared storage across multiple nodes and can be deployed on Kubernetes using the Rook operator. CephFS supports dynamic volume provisioning through the Container Storage Interface (CSI) and can be extended using NFS-Ganesha to provide NFS access for external clients.

This article explains how to deploy CephFS on a Kubernetes Engine cluster using Rook. It covers installing the Rook operator, creating a Ceph cluster, configuring a CephFS filesystem, provisioning persistent volumes, and exporting storage over NFS. It also covers verifying storage functionality, configuring snapshots and monitoring, and preparing the environment for workload integration.

Understanding CephFS Architecture

CephFS maps directly to Google Cloud Filestore features while providing a self-managed and Kubernetes-native storage platform.

GCP Filestore	CephFS Equivalent	Description
Filestore Instance	CephFilesystem	Distributed shared filesystem managed by Rook and Ceph
Filestore NFS Export	CephNFS + NFS-Ganesha	Exposes CephFS storage using the standard NFS protocol
Filestore NFSv3/v4.1	NFS-Ganesha NFSv4	Provides NFS client compatibility for Linux systems and applications
Filestore Capacity Tiers	Ceph pools + StorageClasses	Storage performance and placement controlled through Ceph pool settings
Filestore Snapshots	VolumeSnapshot	CSI-based point-in-time snapshots for CephFS volumes
Filestore Backups	Snapshot export to Object Storage	Backup workflows using snapshots and external object storage
Filestore Multi-share	CephFS subvolumes	Multiple isolated shared volumes provisioned dynamically through CSI
Filestore Performance Tiers	OSD device classes	SSD or HDD-backed OSDs with custom pool tuning for workload optimization
Cloud Monitoring	Ceph Dashboard + Prometheus + Grafana	Integrated monitoring, metrics collection, and visualization stack

Key components of the CephFS deployment include:

Rook Operator: Deploys and manages Ceph services inside the Kubernetes cluster.
Ceph Monitor (MON) and Manager (MGR) Services: Maintain cluster health, quorum, and monitoring services.
Object Storage Daemons (OSDs): Store Ceph data on attached block storage devices.
Metadata Server (MDS) Pods: Manage CephFS metadata and coordinate filesystem access.
Ceph CSI Driver: Dynamically provisions CephFS storage volumes for Kubernetes workloads.
NFS-Ganesha: Exports CephFS storage over NFS for external clients.
Prometheus and Grafana: Monitor Ceph cluster health, storage usage, and filesystem activity.

Prerequisites

Before you begin, you need to:

Deploy a Kubernetes Engine cluster with at least 3 worker nodes, each attached to a raw unformatted Block Storage volume.
Install kubectl and configure access to the Kubernetes cluster.
Install Helm 3 on your management workstation.
Have basic knowledge of Kubernetes storage concepts.

Install the Rook Ceph Operator

Rook extends Kubernetes with orchestration capabilities for Ceph storage services such as MONs, MGRs, OSDs, and MDSs.

Create the project directory.
console
```
$ mkdir -p ~/rook-ceph
```
Switch to the project directory.
console
```
$ cd ~/rook-ceph
```

Verify that all Kubernetes nodes are in the Ready state.

                            console
                            
$ kubectl get nodes

Output:

NAME                         STATUS   ROLES    AGE   VERSION
cephfs-worker-0166bfafa883   Ready    <none>   13m   v1.35.2
cephfs-worker-375d99e1c630   Ready    <none>   14m   v1.35.2
cephfs-worker-3a9f3e64fc10   Ready    <none>   14m   v1.35.2

Add the Rook Helm repository.

                            console
                            
$ helm repo add rook-release https://charts.rook.io/release

Update the local Helm repository index.
console
```
$ helm repo update
```
Install the Rook Ceph operator using the Rook Ceph Helm chart.
console
```
$ helm install rook-ceph rook-release/rook-ceph --namespace rook-ceph --create-namespace --version v1.19.7
```
The command installs the Rook operator and the required Kubernetes custom resource definitions (CRDs).

Note
This article pins the Rook operator to v1.19.7, a tested release that automatically provisions the Ceph CSI driver used for dynamic CephFS volume provisioning. If you install a newer Rook version, confirm that the rook-ceph.cephfs.csi.ceph.com CSI driver is created before provisioning PersistentVolumeClaims.

Verify that the Rook Ceph operator resources are deploying.

                            console
                            
$ kubectl get pods -n rook-ceph

Output:

NAME                                          READY   STATUS    RESTARTS   AGE
ceph-csi-controller-manager-8bcc6877c-mwtxx   1/1     Running   0          11m
rook-ceph-operator-64bb557454-kdjbt           1/1     Running   0          11m

Verify that the Ceph custom resource definitions (CRDs) are available before deploying the Ceph cluster.
console
```
$ kubectl get crds | grep ceph.rook.io
```

Deploy the Ceph Cluster

The Ceph cluster provides the distributed storage backend used by CephFS. Rook deploys Ceph components as Kubernetes pods and manages storage devices attached to the worker nodes. The cluster is configured through the CephCluster custom resource.

Create the Ceph cluster configuration file.

console

$ nano ceph-cluster.yaml

Add the following configuration to the file:

                            yaml
                            
                        
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v19.2.3
  dataDirHostPath: /var/lib/rook
  mon:
    count: 3
    allowMultiplePerNode: false
  mgr:
    count: 1
    allowMultiplePerNode: false
  dashboard:
    enabled: true
    ssl: false
  storage:
    useAllNodes: true
    useAllDevices: false
    deviceFilter: "^vd[b-z]"
  resources:
    mgr:
      requests:
        cpu: "100m"
        memory: "1Gi"
      limits:
        memory: "2Gi"
    mon:
      requests:
        cpu: "250m"
        memory: "512Mi"
      limits:
        memory: "1Gi"
  healthCheck:
    daemonHealth:
      mon:
        interval: 45s
      osd:
        interval: 60s

Save and close the file.

Note

The deviceFilter: "^vd[b-z]" setting instructs Rook to use only additional VirtIO block storage devices such as vdb, vdc, and vdd when creating Ceph OSDs. This excludes the primary system disk, which is typically attached as vda on cloud-based Kubernetes worker nodes. Verify the available block devices on each node before deployment and update the filter if your storage devices use a different naming scheme.

The configuration:

deploys three Ceph monitor (MON) pods for cluster quorum
deploys one Ceph manager (MGR) pod
enables the Ceph dashboard
stores Ceph data in /var/lib/rook
provisions OSDs using attached block storage devices that match the ^vd[b-z] device filter
configures resource requests and limits for Ceph monitor and manager services

Deploy the Ceph cluster.
console
```
$ kubectl apply -f ceph-cluster.yaml
```
Monitor the Ceph pods during deployment and wait for the monitor (MON), manager (MGR), and OSD pods to initialize.
console
```
$ kubectl get pods -n rook-ceph -w
```
Note
The deployment process may take several minutes while Rook initializes the Ceph monitor (MON), manager (MGR), and object storage daemon (OSD) services. If the pods remain in a pending or failed state for an extended period, refer to the Rook Ceph Common Issues Documentation.

Verify that the OSD pods are running.

                            console
                            
$ kubectl get pods -n rook-ceph | grep osd

Output:

rook-ceph-osd-0-68c9846b4-9862n                                   1/1     Running     0          5m40s
rook-ceph-osd-1-5595c4798d-dwcmr                                  1/1     Running     0          5m40s
rook-ceph-osd-2-689c4f4566-h9cxw                                  1/1     Running     0          5m40s
rook-ceph-osd-prepare-ceph-worker-pool-2fd9cd11d442-89kzl         0/1     Completed   0          5m53s
rook-ceph-osd-prepare-ceph-worker-pool-683ee0e17bef-nz26w         0/1     Completed   0          5m52s
rook-ceph-osd-prepare-ceph-worker-pool-aac2507a4c09-8sgdr         0/1     Completed   0          5m51s

Verify that the Ceph cluster reaches the Ready state.

                            console
                            
$ kubectl get cephcluster -n rook-ceph

Output:

NAME        DATADIRHOSTPATH   MONCOUNT   AGE     PHASE   MESSAGE                        HEALTH      EXTERNAL   FSID
rook-ceph   /var/lib/rook     3          8m52s   Ready   Cluster created successfully   HEALTH_OK              c9a844ce-fec1-4dbb-8f07-15144bfd8e1b

Create a CephFS Filesystem

The CephFS filesystem provides shared file storage backed by the Ceph cluster. Rook deploys Ceph metadata server (MDS) pods to manage filesystem metadata and coordinate file access between Kubernetes workloads. The filesystem is defined through the CephFilesystem custom resource.

Create the Ceph filesystem configuration file.

console

$ nano ceph-filesystem.yaml

Add the following configuration to the file:

                            yaml
                            
                        
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: cephfs
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
  dataPools:
    - replicated:
        size: 3
  preserveFilesystemOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true
    resources:
      requests:
        cpu: "100m"
        memory: "256Mi"
      limits:
        memory: "512Mi"

Save and close the file.

The configuration:

creates a CephFS filesystem named cephfs
provisions replicated metadata and data pools with three replicas
preserves the filesystem data if the Kubernetes resource is deleted
deploys one active metadata server (MDS) with one standby instance for failover
configures resource requests and limits for the MDS pods

Deploy the CephFS filesystem.
console
```
$ kubectl apply -f ceph-filesystem.yaml
```
Monitor the CephFS resources during deployment and wait for the MDS pods to initialize.
console
```
$ kubectl get pods -n rook-ceph -w
```

Verify that the MDS pods are running.

                            console
                            
$ kubectl get pods -n rook-ceph | grep mds

Output:

rook-ceph-mds-cephfs-a-77f867b6b8-zzl64                           1/1     Running     0               104s
rook-ceph-mds-cephfs-b-58d447d74f-ft9q5                           1/1     Running     0               103s

Verify that the Ceph filesystem reaches the Ready state.

                            console
                            
$ kubectl get cephfilesystem -n rook-ceph

Output:

NAME     ACTIVEMDS   AGE     PHASE
cephfs   1           2m14s   Ready

Provision CephFS Volumes for Kubernetes

The CephFS CSI driver enables Kubernetes workloads to dynamically provision and mount shared CephFS storage volumes using PersistentVolumeClaims (PVCs).

Create the CephFS StorageClass configuration file.

console

$ nano cephfs-storageclass.yaml

Add the following configuration to the file:

                            yaml
                            
                        
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cephfs-storage
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: cephfs
  pool: cephfs-data0
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true

Save and close the file.

The configuration:

creates a Kubernetes StorageClass named cephfs-storage
uses the Rook CephFS CSI driver for dynamic volume provisioning
provisions storage from the cephfs filesystem
enables dynamic volume expansion
automatically deletes persistent volumes when the associated PVC is removed

Deploy the StorageClass.

                            console
                            
$ kubectl apply -f cephfs-storageclass.yaml

Verify that the StorageClass is created.

                            console
                            
$ kubectl get storageclass

Output:

NAME                             PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
cephfs-storage                   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate           true                   49s
vultr-block-storage (default)    block.csi.vultr.com             Delete          Immediate           true                   4h47m
vultr-block-storage-hdd          block.csi.vultr.com             Delete          Immediate           true                   4h47m
vultr-block-storage-hdd-retain   block.csi.vultr.com             Retain          Immediate           true                   4h47m
vultr-block-storage-retain       block.csi.vultr.com             Retain          Immediate           true                   4h47m
vultr-vfs-storage                block.csi.vultr.com             Delete          Immediate           true                   4h47m
vultr-vfs-storage-retain         block.csi.vultr.com             Retain          Immediate           true                   4h47m

Create the PersistentVolumeClaim configuration file.
console
```
$ nano cephfs-pvc.yaml
```
Add the following configuration to the file:
yaml
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: cephfs-storage
  resources:
    requests:
      storage: 5Gi
```
Save and close the file.

The configuration:
- creates a PersistentVolumeClaim named cephfs-pvc
- provisions a 5Gi CephFS-backed storage volume
- enables multi-node read and write access using the ReadWriteMany access mode
Deploy the PersistentVolumeClaim.
console
```
$ kubectl apply -f cephfs-pvc.yaml
```

Verify that the PersistentVolumeClaim is bound.

                            console
                            
$ kubectl get pvc

Output:

NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
cephfs-pvc   Bound    pvc-bb4b78dd-140f-4b77-90eb-37c60a5134a8   5Gi        RWX            cephfs-storage   <unset>                 1h

Create the test pod configuration file.

console

$ nano cephfs-test-pod.yaml

Add the following configuration to the file:

                            yaml
                            
                        
apiVersion: v1
kind: Pod
metadata:
  name: cephfs-test-pod
spec:
  containers:
    - name: cephfs-test-container
      image: busybox
      command: ["/bin/sh", "-c"]
      args:
        - while true; do sleep 30; done
      volumeMounts:
        - name: cephfs-storage
          mountPath: /mnt/cephfs
  volumes:
    - name: cephfs-storage
      persistentVolumeClaim:
        claimName: cephfs-pvc

Save and close the file.

The configuration mounts the CephFS-backed PersistentVolumeClaim inside the pod at /mnt/cephfs.

Deploy the test pod.

                            console
                            
$ kubectl apply -f cephfs-test-pod.yaml

Verify that the test pod is running.

                            console
                            
$ kubectl get pod cephfs-test-pod

Output:

NAME              READY   STATUS    RESTARTS   AGE
cephfs-test-pod   1/1     Running   0          53s

Create a test file inside the mounted CephFS volume.
console
```
$ kubectl exec -it cephfs-test-pod -- sh
```
Inside the container, create a test file.
console
```
$ echo "CephFS volume test successful" > /mnt/cephfs/test.txt
```

Verify that the file exists.

                            console
                            
$ kubectl exec -it cephfs-test-pod -- cat /mnt/cephfs/test.txt

Output:

CephFS volume test successful

Export CephFS via NFS

The Ceph NFS service enables external systems and legacy applications to access CephFS storage using the standard NFS protocol. Rook deploys NFS-Ganesha pods to export CephFS shares over the network. The NFS service is defined through the CephNFS custom resource, and exports are managed with the ceph nfs export commands.

Create the Ceph NFS configuration file.
console
```
$ nano ceph-nfs.yaml
```
Add the following configuration to the file:
yaml
```
apiVersion: ceph.rook.io/v1
kind: CephNFS
metadata:
  name: ceph-nfs
  namespace: rook-ceph
spec:
  rados:
    pool: .nfs
  server:
    active: 1
    resources:
      requests:
        cpu: "100m"
        memory: "256Mi"
      limits:
        memory: "512Mi"
```
Save and close the file.

The configuration:
- deploys an NFS-Ganesha server named ceph-nfs
- creates a dedicated .nfs Reliable Autonomous Distributed Object Store (RADOS) pool for NFS recovery data
- deploys one active NFS server pod
- configures resource requests and limits for the NFS service
Deploy the NFS service.
console
```
$ kubectl apply -f ceph-nfs.yaml
```
Monitor the NFS resources during deployment.
console
```
$ kubectl get pods -n rook-ceph -w
```

Verify that the NFS-Ganesha pod is running.

                            console
                            
$ kubectl get pods -n rook-ceph | grep nfs

Output:

rook-ceph-nfs-ceph-nfs-a-6bdf6c79c6-c47lr                         2/2     Running            0               48s

Create a Rook Ceph toolbox pod to manage Ceph NFS exports.

                            console
                            
$ kubectl apply -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/toolbox.yaml

Verify that the toolbox pod is running.

                            console
                            
$ kubectl get pods -n rook-ceph | grep tools

Output:

rook-ceph-tools-7c66d65f8b-lh6mt                                  1/1     Running     0                 100s

Create an NFS export for the CephFS filesystem.

                            console
                            
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export create cephfs ceph-nfs /cephfs cephfs --path=/

Output:

{
  "bind": "/cephfs",
  "cluster": "ceph-nfs",
  "fs": "cephfs",
  "mode": "RW",
  "path": "/"
}

The command exports the cephfs filesystem through the ceph-nfs NFS-Ganesha service using the /cephfs export path.

Verify that the NFS export is created.

                            console
                            
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export ls ceph-nfs

Output:

[
  "/cephfs"
]

Retrieve the NFS service details.

                            console
                            
$ kubectl get svc -n rook-ceph | grep nfs

Output:

rook-ceph-nfs-ceph-nfs-a   ClusterIP   10.102.91.113    <none>        2049/TCP,9587/TCP   34m

Mount the NFS Export from a Linux Client

The NFS service is created with a ClusterIP address, which is reachable from the Kubernetes worker nodes and from pods running in the cluster. Mount the exported NFS share from one of the worker nodes.

Note

A ClusterIP address is not reachable from outside the Kubernetes cluster network. To mount the export from a system outside the cluster, such as a separate virtual machine, expose the NFS service using a LoadBalancer or NodePort service instead of the default ClusterIP.

Create a temporary mount directory.
console
```
$ sudo mkdir -p /mnt/cephfs-nfs
```
Install the NFS client utilities.
- On Ubuntu and Debian:
  console
  $ sudo apt install nfs-common -y
- On Rocky Linux and AlmaLinux:
  console
  $ sudo dnf install nfs-utils -y
Mount the exported CephFS NFS share. Replace NFS-SERVICE-IP with the ClusterIP address of the rook-ceph-nfs-ceph-nfs-a service.
console
```
$ sudo mount -t nfs NFS-SERVICE-IP:/cephfs /mnt/cephfs-nfs
```

Create a test file in the mounted NFS share.

                            console
                            
$ echo "NFS export test successful" | sudo tee /mnt/cephfs-nfs/test.txt

Verify that the file exists.
console
```
$ cat /mnt/cephfs-nfs/test.txt
```
Output:
```
NFS export test successful
```
This confirms that the CephFS filesystem is exported through NFS-Ganesha and accessible using the standard NFS protocol.

Configure Storage Pools and Performance

Ceph storage pools control how data is distributed and replicated across the cluster. In addition to the pools that back the CephFS filesystem deployed earlier, you can create dedicated block pools for other Kubernetes workloads and configure replica placement policies to improve resiliency.

Create the Ceph block pool configuration file.
console
```
$ nano ceph-blockpool.yaml
```
Add the following configuration to the file:
yaml
```
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: app-block-pool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
  parameters:
    compression_mode: none
```
Save and close the file.

The configuration:
- creates a replicated Ceph block pool named app-block-pool, independent of the pools backing the CephFS filesystem, for workloads that need RBD-backed block storage
- stores three copies of each object across different Kubernetes worker nodes
- uses the host failure domain to distribute replicas across separate nodes
- disables compression for predictable storage performance
Deploy the storage pool.
console
```
$ kubectl apply -f ceph-blockpool.yaml
```

Verify that the storage pool is created.

                            console
                            
$ kubectl get cephblockpool -n rook-ceph

Output:

NAME              PHASE   TYPE         FAILUREDOMAIN   AGE
app-block-pool    Ready   Replicated   host            11s

View the Ceph cluster health.

                            console
                            
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status

Output:

cluster:
    id:     c9a844ce-fec1-4dbb-8f07-15144bfd8e1b
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 21h)
    mgr: a(active, since 9s)
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 21h), 3 in (since 21h)

  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 35 objects, 1.3 MiB
    usage:   248 MiB used, 304 GiB / 304 GiB avail
    pgs:     113 active+clean

  io:
    client:   26 KiB/s rd, 0 B/s wr, 10 op/s rd, 1 op/s wr

This confirms that the Ceph block pool is configured and the cluster is operating in a healthy state.

Set Up Snapshots and Backups

CephFS snapshots allow you to capture point-in-time copies of persistent volumes and restore data when needed. Kubernetes uses the CSI snapshot controller together with the Rook CephFS CSI driver to manage volume snapshot operations.

Install the Kubernetes CSI Snapshot CRDs

The Kubernetes CSI snapshot CRDs enable Kubernetes to manage volume snapshots for CSI-compatible storage drivers such as Rook CephFS.

Deploy the VolumeSnapshotClass CRD.

                            console
                            
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshotclasses.yaml

Deploy the VolumeSnapshotContent CRD.

                            console
                            
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshotcontents.yaml

Deploy the VolumeSnapshot CRD.

                            console
                            
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/client/config/crd/snapshot.storage.k8s.io_volumesnapshots.yaml

Deploy the snapshot controller RBAC resources.

                            console
                            
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/deploy/kubernetes/snapshot-controller/rbac-snapshot-controller.yaml

Deploy the snapshot controller.

                            console
                            
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/release-8.2/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

Verify that the snapshot controller pods are running.

                            console
                            
$ kubectl get pods -n kube-system | grep snapshot

Output:

snapshot-controller-74c7dc46c-c7xbv        1/1     Running   0          33s
snapshot-controller-74c7dc46c-lmdlv        1/1     Running   0          33s

Create a VolumeSnapshotClass

The VolumeSnapshotClass defines how Kubernetes creates and manages snapshots using the Rook CephFS CSI driver.

Create the snapshot class configuration file.
console
```
$ nano cephfs-snapshotclass.yaml
```
Add the following configuration to the file:
yaml
```
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: cephfs-snapshotclass
driver: rook-ceph.cephfs.csi.ceph.com
deletionPolicy: Delete
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph
```
Save and close the file.

The configuration:
- creates a snapshot class named cephfs-snapshotclass
- uses the Rook CephFS CSI driver for snapshot operations
- configures the CephFS snapshotter secret used by the CSI driver
- automatically deletes snapshots when the snapshot resource is removed

Deploy the snapshot class.

                            console
                            
$ kubectl apply -f cephfs-snapshotclass.yaml

Verify that the snapshot class is created.

                            console
                            
$ kubectl get volumesnapshotclass

Output:

NAME                   DRIVER                          DELETIONPOLICY   AGE
cephfs-snapshotclass   rook-ceph.cephfs.csi.ceph.com   Delete           12s

Create a CephFS Volume Snapshot

The VolumeSnapshot resource creates a point-in-time snapshot of a PersistentVolumeClaim using the CephFS CSI driver.

Create the snapshot configuration file.
console
```
$ nano cephfs-snapshot.yaml
```
Add the following configuration to the file:
yaml
```
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: cephfs-snapshot
spec:
  volumeSnapshotClassName: cephfs-snapshotclass
  source:
    persistentVolumeClaimName: cephfs-pvc
```
Save and close the file.

The configuration creates a snapshot of the cephfs-pvc PersistentVolumeClaim using the cephfs-snapshotclass snapshot class.

Deploy the snapshot.

                            console
                            
$ kubectl apply -f cephfs-snapshot.yaml

Verify that the snapshot is ready.

                            console
                            
$ kubectl get volumesnapshot

Output:

NAME                  READYTOUSE   SOURCEPVC    SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS          SNAPSHOTCONTENT                                    CREATIONTIME   AGE
cephfs-snapshot       true         cephfs-pvc                           5Gi           cephfs-snapshotclass   snapcontent-06f8cb1b-a46a-4de9-9aa8-eb7ddd34aa04   46s            47s

Restore a Snapshot to a New PersistentVolumeClaim

You can restore a CephFS snapshot to a new PersistentVolumeClaim to recover application data or create a duplicate volume from an existing snapshot.

Create the restore PVC configuration file.
console
```
$ nano cephfs-restore-pvc.yaml
```
Add the following configuration to the file:
yaml
```
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cephfs-restore-pvc
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: cephfs-storage
  resources:
    requests:
      storage: 5Gi
  dataSource:
    name: cephfs-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
```
Save and close the file.

The configuration:
- creates a new PersistentVolumeClaim named cephfs-restore-pvc
- restores data from the cephfs-snapshot snapshot
- uses the cephfs-storage StorageClass for CephFS storage
- provisions a new 5Gi restored volume from the snapshot source
Note
The restored PersistentVolumeClaim size must be equal to or larger than the source snapshot restore size. Kubernetes does not allow restoring snapshots into smaller volumes.
Deploy the restored PersistentVolumeClaim.
console
```
$ kubectl apply -f cephfs-restore-pvc.yaml
```

Verify that the restored PersistentVolumeClaim is bound.

                            console
                            
$ kubectl get pvc

Output:

NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
cephfs-pvc           Bound    pvc-bb4b78dd-140f-4b77-90eb-37c60a5134a8   5Gi        RWX            cephfs-storage   <unset>                 40h
cephfs-restore-pvc   Bound    pvc-338939cf-72a3-45b9-a978-31a54e29ead4   5Gi        RWX            cephfs-storage   <unset>                 31s

Configure Scheduled CephFS Snapshots

Kubernetes does not create scheduled snapshots automatically. You can use a Kubernetes CronJob resource to create recurring CephFS snapshots for backup and recovery workflows. The CronJob runs kubectl inside the cluster, so it requires a ServiceAccount with role-based access control (RBAC) permission to manage VolumeSnapshot resources.

Create the snapshot RBAC configuration file.

console

$ nano cephfs-snapshot-rbac.yaml

Add the following configuration to the file:

                            yaml
                            
                        
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cephfs-snapshot-sa
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-snapshot-role
  namespace: default
rules:
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-snapshot-rolebinding
  namespace: default
subjects:
  - kind: ServiceAccount
    name: cephfs-snapshot-sa
    namespace: default
roleRef:
  kind: Role
  name: cephfs-snapshot-role
  apiGroup: rbac.authorization.k8s.io

Save and close the file.

The configuration:

creates a ServiceAccount named cephfs-snapshot-sa for the snapshot CronJob
grants a Role that allows managing VolumeSnapshot resources in the default namespace
binds the Role to the ServiceAccount

Deploy the snapshot RBAC resources.
console
```
$ kubectl apply -f cephfs-snapshot-rbac.yaml
```

Create the scheduled snapshot configuration file.

console

$ nano cephfs-snapshot-cronjob.yaml

Add the following configuration to the file:

                            yaml
                            
                        
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cephfs-snapshot-cronjob
spec:
  schedule: "0 */6 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: cephfs-snapshot-sa
          restartPolicy: OnFailure
          containers:
            - name: snapshot-creator
              image: alpine/k8s:1.36.2
              command:
                - /bin/sh
                - -c
                - |
                  SNAPSHOT_NAME="cephfs-snapshot-$(date +%s)"
                  cat <<EOF | kubectl apply -f -
                  apiVersion: snapshot.storage.k8s.io/v1
                  kind: VolumeSnapshot
                  metadata:
                    name: ${SNAPSHOT_NAME}
                  spec:
                    volumeSnapshotClassName: cephfs-snapshotclass
                    source:
                      persistentVolumeClaimName: cephfs-pvc
                  EOF

Save and close the file.

The configuration:

creates a Kubernetes CronJob named cephfs-snapshot-cronjob
runs under the cephfs-snapshot-sa ServiceAccount with permission to manage snapshots
creates a new CephFS snapshot every 6 hours
generates unique snapshot names using timestamps
uses the cephfs-snapshotclass snapshot class
creates snapshots of the cephfs-pvc PersistentVolumeClaim

Deploy the scheduled snapshot CronJob.
console
```
$ kubectl apply -f cephfs-snapshot-cronjob.yaml
```

Verify that the CronJob is created.

                            console
                            
$ kubectl get cronjob

Output:

NAME                      SCHEDULE      TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cephfs-snapshot-cronjob   0 */6 * * *   <none>     False     0        <none>          10s

Configure Authentication and Access Control

CephFS uses CephX authentication to control client access to the filesystem. You can create restricted CephFS users, limit access to specific directories, and apply POSIX file permissions to isolate workloads between tenants and applications.

Configure POSIX File Permissions

CephFS honors standard POSIX ownership and permission bits, so you can isolate directories for different tenants using the same chown and chmod commands you would use on a local filesystem.

Open a shell inside the test pod.
console
```
$ kubectl exec -it cephfs-test-pod -- sh
```
Create separate directories for different tenants inside the mounted CephFS volume.
console
```
$ mkdir -p /mnt/cephfs/tenant-a /mnt/cephfs/tenant-b
```
Assign ownership to the tenant-a directory.
console
```
$ chown 1001:1001 /mnt/cephfs/tenant-a
```
Restrict access to the directory owner only.
console
```
$ chmod 700 /mnt/cephfs/tenant-a
```

Verify the directory permissions.

                            console
                            
$ ls -ld /mnt/cephfs/tenant-a

Output:

drwx------    2 1001     1001             0 May 28 00:12 /mnt/cephfs/tenant-a

Exit the pod shell.
console
```
$ exit
```

Create Restricted CephFS Client Users

CephFS supports path-based authorization using CephX authentication. You can create dedicated CephFS users and restrict access to specific filesystem paths for tenant isolation and workload security.

Open a shell inside the Rook toolbox pod.

                            console
                            
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- bash

Create a CephFS client user with read and write access restricted to the /tenant-a directory.
console
```
$ ceph fs authorize cephfs client.tenant-a /tenant-a rw
```
Retrieve the authentication key for the client.tenant-a CephFS user.
console
```
$ ceph auth get-key client.tenant-a
```
View the client capabilities.
console
```
$ ceph auth get client.tenant-a
```
Exit the toolbox shell.
console
```
$ exit
```
Note
The path passed to ceph fs authorize is resolved from the root of the CephFS filesystem, not from the pod's local PersistentVolumeClaim mount point. If the CSI driver provisions the PersistentVolumeClaim as a subvolume rather than exposing the filesystem root directly, retrieve the subvolume's actual path with ceph fs subvolume getpath before authorizing access, and adjust the path argument to match.
Verify that the client.tenant-a credentials restrict access to the intended path. Mount CephFS directly on a Linux client using the kernel client and the retrieved key. Replace MON-IP with the IP address of a Ceph monitor, and CLIENT-KEY with the key retrieved for client.tenant-a in the previous step.
console
```
$ sudo mount -t ceph MON-IP:6789:/tenant-a /mnt/tenant-a-test -o name=tenant-a,secret=CLIENT-KEY
```
Confirm that the mount succeeds and that only the authorized directory is visible.
console
```
$ ls /mnt/tenant-a-test
```
A permission error when accessing paths outside the mounted directory confirms that the CephX authorization correctly restricts the client's access.

Set Up Monitoring

Ceph provides built-in monitoring features through the Ceph Dashboard, Prometheus metrics endpoints, and Grafana dashboards. You can use these services to monitor cluster health, storage utilization, CephFS activity, and OSD performance.

Verify that the Ceph Dashboard Is Enabled

The Ceph Manager (MGR) service hosts the Ceph Dashboard and monitoring modules.

Verify that the Ceph cluster is healthy.

                            console
                            
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status

Output:

  cluster:
    id:     c9a844ce-fec1-4dbb-8f07-15144bfd8e1b
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 47h)
    mgr: a(active, since 3h)
    mds: 1/1 daemons up, 1 hot standby
    osd: 3 osds: 3 up (since 47h), 3 in (since 47h)

  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 46 objects, 3.9 MiB
    usage:   502 MiB used, 304 GiB / 304 GiB avail
    pgs:     113 active+clean

  io:
    client:   938 B/s rd, 1 op/s rd, 0 op/s wr

Verify that the Ceph Manager pod is running.
console
```
$ kubectl get pods -n rook-ceph | grep mgr
```

Retrieve the Ceph Dashboard service details.

                            console
                            
$ kubectl get svc -n rook-ceph | grep dashboard

Output:

rook-ceph-mgr-dashboard    ClusterIP   10.111.11.76     <none>        7000/TCP            47h

Access the Ceph Dashboard

The Ceph Dashboard service defaults to a ClusterIP address, so reach it from your local workstation using a kubectl port-forward session.

Forward the Ceph Dashboard service to your local workstation.
console
```
$ kubectl port-forward -n rook-ceph svc/rook-ceph-mgr-dashboard 7000:7000
```
Open http://127.0.0.1:7000 in a web browser.
Retrieve the dashboard administrator password. Open a new terminal session and run the following command.
console
```
$ kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
```
Copy the password from the command output.
Use the default username admin and paste the copied password to log in to the Ceph Dashboard.

Enable the Prometheus Monitoring Module

Ceph exports monitoring metrics through the Prometheus module running on the Ceph Manager service.

Enable the Prometheus module.

                            console
                            
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph mgr module enable prometheus

Verify that the Prometheus module is enabled.

                            console
                            
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph mgr module ls | grep prometheus

Retrieve the Ceph Manager service details.
console
```
$ kubectl get svc -n rook-ceph | grep mgr
```

Deploy Grafana Dashboards

Grafana visualizes Prometheus metrics using dashboards for Ceph cluster monitoring.

Add the Grafana Community Helm repository.

                            console
                            
$ helm repo add grafana-community https://grafana-community.github.io/helm-charts

Update the Helm repository index.
console
```
$ helm repo update
```

Deploy Grafana.

                            console
                            
$ helm install grafana grafana-community/grafana --namespace monitoring --create-namespace

Verify that the Grafana pod is running.

                            console
                            
$ kubectl get pods -n monitoring

Output:

NAME                       READY   STATUS    RESTARTS   AGE
grafana-5786bf9b45-jml9z   1/1     Running   0          3m9s

Retrieve the Grafana administrator password.

                            console
                            
$ kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Copy the password value.

Forward the Grafana service to your local workstation.
console
```
$ kubectl port-forward -n monitoring svc/grafana 3000:80
```
Open http://127.0.0.1:3000 in a web browser.
Use the default username admin and the password retrieved in Step 5 to log in to Grafana.

Verify the Deployment

You can perform an end-to-end validation to verify the CephFS deployment, NFS export, snapshot functionality, and monitoring services.

Verify CephFS PersistentVolumeClaim Access

Confirm that the PersistentVolumeClaim provisioned earlier remains bound and that the test pod can still read and write to the mounted CephFS volume.

Verify that the CephFS-backed PersistentVolumeClaim is bound.

                            console
                            
$ kubectl get pvc

Output:

NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
cephfs-pvc           Bound    pvc-bb4b78dd-140f-4b77-90eb-37c60a5134a8   5Gi        RWX            cephfs-storage   <unset>                 44h
cephfs-restore-pvc   Bound    pvc-338939cf-72a3-45b9-a978-31a54e29ead4   5Gi        RWX            cephfs-storage   <unset>                 4h32m

Verify that the CephFS test pod is running.

                            console
                            
$ kubectl get pod cephfs-test-pod

Output:

NAME              READY   STATUS    RESTARTS   AGE
cephfs-test-pod   1/1     Running   0          31h

Create a validation file inside the mounted CephFS volume.

                            console
                            
$ kubectl exec -it cephfs-test-pod -- sh -c 'echo "CephFS deployment verification successful" > /mnt/cephfs/verify.txt'

Verify that the file exists.

                            console
                            
$ kubectl exec -it cephfs-test-pod -- cat /mnt/cephfs/verify.txt

Output:

CephFS deployment verification successful

Verify the CephFS NFS Export

Confirm that the NFS-Ganesha export created earlier is still registered and that the underlying service remains reachable.

Verify that the NFS export is available through the Ceph NFS service.

                            console
                            
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph nfs export ls ceph-nfs

Output:

[
  "/cephfs"
]

Verify that the NFS service is running.

                            console
                            
$ kubectl get svc -n rook-ceph | grep nfs

Output:

rook-ceph-nfs-ceph-nfs-a   ClusterIP   10.102.91.113    <none>        2049/TCP,9587/TCP   30h

Verify Snapshot and Restore Operations

Confirm that the snapshot and restore workflow completed successfully and that both the original and restored volumes remain available.

Verify that the CephFS snapshot is ready.
console
```
$ kubectl get volumesnapshot
```

Verify that the restored PersistentVolumeClaim is bound.

                            console
                            
$ kubectl get pvc cephfs-restore-pvc

Output:

NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     VOLUMEATTRIBUTESCLASS   AGE
cephfs-restore-pvc   Bound    pvc-338939cf-72a3-45b9-a978-31a54e29ead4   5Gi        RWX            cephfs-storage   <unset>                 4h43m

Verify Monitoring Services

Confirm that the Ceph Dashboard and Grafana remain available and that the cluster continues reporting a healthy status after the storage operations performed throughout this article.

Verify that the Ceph Dashboard service is running.
console
```
$ kubectl get svc -n rook-ceph | grep dashboard
```

Verify that the Grafana pod is running.

                            console
                            
$ kubectl get pods -n monitoring

Output:

NAME                       READY   STATUS    RESTARTS   AGE
grafana-5786bf9b45-jml9z   1/1     Running   0          47m

Verify that the Ceph cluster remains healthy after all storage operations.
console
```
$ kubectl exec -it -n rook-ceph deploy/rook-ceph-tools -- ceph status
```

Migrate from GCP Filestore to CephFS

This section outlines the strategy for migrating workloads from GCP Filestore to CephFS on Kubernetes. It focuses on the concepts and the tools involved so the approach stays applicable as environments and tooling versions change. Consult the current GCP Filestore documentation and the Rook and Ceph CephFS documentation for exact commands and options when you plan your migration.

Data Migration

Data migration copies the existing data set from the Filestore NFS export to CephFS while preserving file permissions and ownership. Because both Filestore and the CephFS NFS-Ganesha export present a standard NFS interface, a client that mounts both file systems can transfer data between them with a general-purpose tool such as rsync. Mount the Filestore export and the CephFS export on the same client, then run rsync in archive mode (rsync -avh) so it preserves ownership, permissions, symbolic links, and timestamps during the copy. For large data sets, parallelize the transfer by running rsync from multiple clients against different subdirectories, or copy the data into a pod that has the CephFS PersistentVolumeClaim mounted.

Application Migration

Application migration redirects Kubernetes workloads from Filestore-backed storage to CephFS. Because CephFS provides the same POSIX and ReadWriteMany semantics that Filestore offers, most applications continue to work without code changes. Update each workload's PersistentVolumeClaim to reference the cephfs-storage StorageClass instead of the Filestore StorageClass (for example, filestore-rwx), then recreate the PersistentVolumeClaim and restart the workload so it binds to the new CephFS volume. For applications that run outside Kubernetes, repoint the NFS mount target at the CephFS NFS-Ganesha export instead of the Filestore endpoint, using the same mount -t nfs command shown earlier in this article.

Snapshot and Backup Migration

Kubernetes CSI VolumeSnapshot resources are the CephFS equivalent of Filestore snapshots. Once the CSI snapshot controller and the cephfs-snapshotclass are in place, point-in-time snapshots of any CephFS PersistentVolumeClaim are created by referencing the claim from a VolumeSnapshot resource, as shown earlier in this article. If the existing environment uses scheduled Filestore backups, recreate the schedules with a Kubernetes CronJob or an external backup automation tool, and store long-term copies in an object storage backend.

Data Storage Considerations

GCP Filestore manages capacity expansion automatically, while CephFS requires manual OSD expansion and replication planning as storage usage grows. Plan capacity across the Ceph cluster based on expected workload patterns.

Replication overhead: CephFS uses a replication factor of 3 by default, so raw storage usage is roughly three times the usable capacity. Account for this multiplier when sizing the OSD devices.
Adding capacity: Attach additional Block Storage volumes to the worker nodes and let Rook provision new OSDs through the device filter. Ceph rebalances data across the expanded set automatically.
Pool placement: Use separate Ceph pools or OSD device classes to place performance-sensitive workloads on faster storage, similar to Filestore capacity tiers.

Things to Consider During Migration

The following items require attention during the migration window:

NFS protocol compatibility: Some Filestore deployments use NFSv3 clients. CephFS exports through NFS-Ganesha use NFSv4 by default. Verify client compatibility before migration.
Storage performance tuning: Filestore performance tiers are managed automatically. CephFS performance depends on OSD device type, replication settings, network throughput, and Ceph pool configuration.
Multi-share migration: Filestore multi-share environments map to CephFS subdirectories or separate CephFS subvolumes for tenant isolation.
Authentication and access control: GCP Filestore integrates with Google Cloud Identity and Access Management (IAM) policies. CephFS uses CephX authentication together with POSIX permissions and ACLs.
Monitoring migration: Replace Google Cloud Monitoring dashboards with the Ceph Dashboard, Prometheus metrics, and Grafana dashboards.
Capacity planning: CephFS replication increases raw storage usage compared to single-copy storage systems. Plan storage growth before migrating production workloads.

Conclusion

You have deployed CephFS on a Kubernetes Engine cluster using the Rook operator and configured shared persistent storage for Kubernetes workloads. The setup includes dynamic CephFS volume provisioning, NFS exports using NFS-Ganesha, CSI snapshot and restore operations, CephX authentication, monitoring with the Ceph Dashboard and Grafana, and migration guidance from GCP Filestore. This configuration provides a scalable and self-managed distributed storage platform for containerized applications running on Kubernetes. For additional configuration options and production best practices, refer to the Rook documentation and Ceph documentation.

Tags:

Open Source

GCP

How to Deploy CephFS as a GCP Filestore Alternative

How to Deploy CephFS as a GCP Filestore Alternative

Understanding CephFS Architecture

Prerequisites

Install the Rook Ceph Operator

Deploy the Ceph Cluster

Create a CephFS Filesystem

Provision CephFS Volumes for Kubernetes

Export CephFS via NFS

Mount the NFS Export from a Linux Client

Configure Storage Pools and Performance

Set Up Snapshots and Backups

Install the Kubernetes CSI Snapshot CRDs

Create a VolumeSnapshotClass

Create a CephFS Volume Snapshot

Restore a Snapshot to a New PersistentVolumeClaim

Configure Scheduled CephFS Snapshots

Configure Authentication and Access Control

Configure POSIX File Permissions

Create Restricted CephFS Client Users

Set Up Monitoring

Verify that the Ceph Dashboard Is Enabled

Access the Ceph Dashboard

Enable the Prometheus Monitoring Module

Deploy Grafana Dashboards

Verify the Deployment

Verify CephFS PersistentVolumeClaim Access

Verify the CephFS NFS Export

Verify Snapshot and Restore Operations

Verify Monitoring Services

Migrate from GCP Filestore to CephFS

Data Migration

Application Migration

Snapshot and Backup Migration

Data Storage Considerations

Things to Consider During Migration

Conclusion

Comments

Tech Talks

Vultr Blogs