Deploying Ceph All-in-One Using Cephadm

This post is more like a note to myself and for lab purposes, but this may help if you are looking at How to Deploy Ceph All-in-One using Cephadm.

Env

VM spec.

• Hostname: sandbox
• OS: Ubuntu 22.04
• IP: 10.201.55.10
• 8 vCPU
• 8G RAM
• 35GB vda (OS disk)
• 40GB vdb
• 40GB vdc
• 40GB vdd
• 40GB vde

The cluster will be configured with no replica because cluster has single node only.

Deploy Ceph

Install cephadm & ceph-common

apt install cephadm ceph-common ceph-volume -y
sudo modprobe ceph
sudo modprobe rbd

Bootstrap ceph cluster

cephadm add-repo --release quincy
cephadm install

sudo cephadm bootstrap --mon-ip 10.201.55.10 --initial-dashboard-user admin --initial-dashboard-password gladiators88 --skip-monitoring-stack

ceph -v
ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)

Check available disks

ceph -s
ceph orch device ls
HOST     PATH      TYPE  DEVICE ID   SIZE  AVAILABLE  REFRESHED  REJECT REASONS              
sandbox  /dev/vdb  hdd              40.0G  No         61s ago    Has BlueStore device label  
sandbox  /dev/vdc  hdd              40.0G  No         61s ago    Has BlueStore device label  
sandbox  /dev/vdd  hdd              40.0G  No         61s ago    Has BlueStore device label  
sandbox  /dev/vde  hdd              40.0G  No         61s ago    Has BlueStore device label

Zap & Clean Disk

for i in {b..e}; do sgdisk -Z /dev/vd$i; done

ceph-volume lvm zap /dev/vdb --destroy
ceph-volume lvm zap /dev/vdc --destroy
ceph-volume lvm zap /dev/vdd --destroy
ceph-volume lvm zap /dev/vde --destroy

ceph orch device zap sandbox /dev/vdb --force
ceph orch device zap sandbox /dev/vdc --force
ceph orch device zap sandbox /dev/vdd --force
ceph orch device zap sandbox /dev/vde --force

ceph orch device ls --refresh
HOST     PATH      TYPE  DEVICE ID   SIZE  AVAILABLE  REFRESHED  REJECT REASONS  
sandbox  /dev/vdb  hdd              40.0G  Yes        0s ago                     
sandbox  /dev/vdc  hdd              40.0G  Yes        0s ago                     
sandbox  /dev/vdd  hdd              40.0G  Yes        0s ago                     
sandbox  /dev/vde  hdd              40.0G  Yes        0s ago

Add OSDs

ceph orch apply osd --all-available-devices

root@sandbox:~# ceph -s
  cluster:
    id:     14953fa8-9e84-11ef-a909-31f7d67a1cbd
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum sandbox (age 11m)
    mgr: sandbox.qtrsaa(active, since 11m)
    osd: 4 osds: 4 up (since 4s), 4 in (since 27s)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 0 objects, 0 B
    usage:   1.1 GiB used, 159 GiB / 160 GiB avail
    pgs:     100.000% pgs not active
             1 undersized+peered

root@sandbox:~# ceph osd tree
ID  CLASS  WEIGHT   TYPE NAME         STATUS  REWEIGHT  PRI-AFF
-1         0.15637  root default                               
-3         0.15637      host sandbox                           
 0    hdd  0.03909          osd.0         up   1.00000  1.00000
 1    hdd  0.03909          osd.1         up   1.00000  1.00000
 2    hdd  0.03909          osd.2         up   1.00000  1.00000
 3    hdd  0.03909          osd.3         up   1.00000  1.00000

root@sandbox:~# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP  META     AVAIL    %USE  VAR   PGS  STATUS
 0    hdd  0.03909   1.00000   40 GiB  290 MiB  152 KiB   0 B  290 MiB   40 GiB  0.71  1.00    0      up
 1    hdd  0.03909   1.00000   40 GiB  290 MiB  152 KiB   0 B  290 MiB   40 GiB  0.71  1.00    1      up
 2    hdd  0.03909   1.00000   40 GiB  290 MiB  152 KiB   0 B  290 MiB   40 GiB  0.71  1.00    0      up
 3    hdd  0.03909   1.00000   40 GiB  290 MiB  152 KiB   0 B  290 MiB   40 GiB  0.71  1.00    0      up
                       TOTAL  160 GiB  1.1 GiB  608 KiB   0 B  1.1 GiB  159 GiB  0.71                   
MIN/MAX VAR: 1.00/1.00  STDDEV: 0

a few minutes later, the ceph status will become HEALTH_WARN. This is due to a pool has default size 3 and min_size 2. But the all-in-one cluster is only able to store single replica.

ceph osd dump | grep 'replicated size'
---
pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr

Configure Cluster to allow pool size one

ceph config set global mon_allow_pool_size_one true
ceph osd pool set .mgr size 2 --yes-i-really-mean-it
set pool 1 size to 2
ceph osd pool set .mgr size 1 --yes-i-really-mean-it
set pool 1 size to 1

# this step is not necessary here as the size already decreased to 1
ceph osd pool set .mgr min_size 1
set pool 1 min_size to 1

ceph osd dump | grep 'replicated size'
---
pool 1 '.mgr' replicated size 1 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr

Enable RBD

I'm using rbd for my kubernetes, so i'll create a pool called kubernetes

ceph osd pool create kubernetes 64 64 replicated
pool 'kubernetes' created

Set size to 1

ceph osd pool set kubernetes min_size 1
set pool 2 min_size to 1

ceph osd pool set kubernetes size 2 --yes-i-really-mean-it
set pool 2 size to 2

ceph osd pool set kubernetes size 1 --yes-i-really-mean-it
set pool 2 size to 1

Config pool for RBD use

rbd pool init kubernetes
ceph osd pool application enable kubernetes rbd
enabled application 'rbd' on pool 'kubernetes'

Create keyring

ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
[client.kubernetes]
        key = AQBUPS9nb66QGBAAgEsRltnrH5ngw9sn0LUwHA==

👍🏻👍🏻

Enable CephFS

Create Pool for cephfs

ceph osd pool create mycephfs_data 32
ceph osd pool create mycephfs_metadata 32
ceph fs new mycephfs mycephfs_metadata mycephfs_data
ceph fs ls
ceph fs status mycephfs

Enable mds

ceph orch apply mds mycephfs --placement="sandbox"
ceph orch ls
ceph orch ps --daemon_type=mds
ceph mds stat

Set size to 1 for both pool data & metadata

ceph osd pool set mycephfs_data min_size 1
set pool 3 min_size to 1
ceph osd pool set mycephfs_data size 2 --yes-i-really-mean-it
set pool 3 size to 2
ceph osd pool set mycephfs_data size 1 --yes-i-really-mean-it
set pool 3 size to 1

ceph osd pool set mycephfs_metadata min_size 1
set pool 4 min_size to 1
ceph osd pool set mycephfs_metadata size 2 --yes-i-really-mean-it
set pool 4 size to 2
ceph osd pool set mycephfs_metadata size 1 --yes-i-really-mean-it
set pool 4 size to 1

Export admin keyring (not best practice)

ceph-authtool -p /etc/ceph/ceph.client.admin.keyring > admin.key
cat admin.key 
AQAsNy9nePJsKBAAKm6XUBu9NzfoV8vcbMHh5Q==

(Optional) Create subvolumegroup for Kubernetes use

ceph fs volume ls
[
    {
        "name": "mycephfs"
    }
]
ceph fs subvolumegroup create mycephfs csi
ceph fs subvolumegroup ls mycephfs
[
    {
        "name": "csi"
    }
]

🤝🏻🤝🏻

Ignore No Replicas Warning

ceph health mute POOL_NO_REDUNDANCY

ceph -s
  cluster:
    id:     14953fa8-9e84-11ef-a909-31f7d67a1cbd
    health: HEALTH_OK
            (muted: POOL_NO_REDUNDANCY)
 
  services:
    mon: 1 daemons, quorum sandbox (age 23m)
    mgr: sandbox.qtrsaa(active, since 22m)
    mds: 1/1 daemons up
    osd: 4 osds: 4 up (since 11m), 4 in (since 12m)
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 129 pgs
    objects: 25 objects, 451 KiB
    usage:   1.1 GiB used, 159 GiB / 160 GiB avail
    pgs:     129 active+clean

Other Config

Allow pool deletion

ceph config set global mon_allow_pool_delete true

Reference

Build all-in-one Ceph cluster via cephadm - HackMD

# Build all-in-one Ceph cluster via cephadm ###### tags: `ceph` Deploy all-in-one ceph cluster for

HackMD