Freeze Openstack & Ceph Cluster - POC Baremetal Server Migration

This article is more like a note to myself, but this may help if you are looking at How to Freeze Openstack & Ceph Cluster - POC Baremetal Server Migration.

❄ Freeze & Shutdown Cluster

1. Stop Ceph & Openstack Cluster

|--------------------------|
| Stop Openstack instances |
|--------------------------|

openstack server stop instance-1a
openstack server stop instance-1b
openstack server stop instance-2a
openstack server stop instance-2b

openstack server list


|--------------------|
| Stop Nova services |
|--------------------|

openstack compute service set --disable --down compute-node01 nova-compute
openstack compute service set --disable --down compute-node02 nova-compute
openstack compute service set --disable --down compute-node03 nova-compute

openstack compute service list


|----------------------|
| Stop Cinder services |
|----------------------|

openstack volume service set --disable node03@rbd-1 cinder-volume
openstack volume service set --disable node02@rbd-1 cinder-volume
openstack volume service set --disable node01@rbd-1 cinder-volume
openstack volume service set --disable node01 cinder-scheduler
openstack volume service set --disable node02 cinder-scheduler
openstack volume service set --disable node03 cinder-scheduler


|--------------------|
| Pause Ceph Cluster |
|--------------------|

ceph osd set norecover
ceph osd set nobackfill
ceph osd set noout
ceph osd set norebalance
ceph osd set pause


|--------------------------------------|
| Notes the machine-id of each servers |
|--------------------------------------|

cat /etc/machine-id

2. Graceful shutdown servers

shutdown -h now

🧳 Move Baremetal Servers

* Make sure to labeling network ports, harddisks, and node name
* Unplug power cord and network cables safely after server fully stopped
* Dont change network configuration beetwen all servers and switches

⚡ Power On & Start Cluster

1. Power On Servers

* Start from node01, node02, then node03
* Check Anydesk for remote access to console and Ngrok dashboard
* Check Ngrok tunnel port

2. Verify Servers running well

cat /etc/machine-id
lsblk
df -hT
ip address show
ip route show
systemctl list-units --type service
docker ps -a

## ping jumbo frame to all nodes and internet.
./ping.sh

3. Start and Enable Ceph Cluster

ceph osd unset norecover
ceph osd unset nobackfill
ceph osd unset noout
ceph osd unset norebalance
ceph osd unset pause

## Verify
ceph -s
ceph health detail

4. Start MariaDB Recovery

ssh root@node01
sudo -i
cd /root/openstack
source /root/openstack/admin-openrc.sh
kolla-ansible -v -i multinode mariadb_recovery

5. Start and Enable Openstack Cluster

|-----------------------|
| Start Cinder services |
|-----------------------|

openstack volume service set --enable node01 cinder-scheduler
openstack volume service set --enable node02 cinder-scheduler
openstack volume service set --enable node03 cinder-scheduler
openstack volume service set --enable node01@rbd-1 cinder-volume
openstack volume service set --enable node02@rbd-1 cinder-volume
openstack volume service set --enable node03@rbd-1 cinder-volume

## Verify
openstack volume service list


|---------------------|
| Start Nova services |
|---------------------|

openstack compute service set --enable --up compute-node01 nova-compute
openstack compute service set --enable --up compute-node02 nova-compute
openstack compute service set --enable --up compute-node03 nova-compute

## Verify
openstack compute service list


|------------------------------|
| Verify All Services Run Well |
|------------------------------|

openstack endpoint list
openstack service list
openstack network agent list
openstack compute service list
openstack volume service list

6. Start Intances

|---------------------|
| Start All Instances |
|---------------------|

openstack server start instance-1a
openstack server start instance-1b
openstack server start instance-2a
openstack server start instance-2b

## Verify
openstack server list --long


|----------------|
| Ping Instances |
|----------------|

ping -c5 192.168.192.123
ping -c5 192.168.192.185
ping -c5 192.168.192.7
ping -c5 192.168.192.126


|---------------|
| SSH Instances |
|---------------|

ssh [email protected] hostname
ssh [email protected] hostname
ssh [email protected] hostname
ssh [email protected] hostname

🌬 Smoke Test After Migration

|--------------------------|
| Create Openstack project |
|--------------------------|

openstack project create project-smoke-test


|-------------------------|
| Create Internal Network |
|-------------------------|

openstack network create internal-net-2

openstack subnet create \
  --network internal-net-2 \
  --subnet-range 11.0.0.0/24 \
  --gateway 11.0.0.1 \
  --dns-nameserver 8.8.8.8 \
  --dns-nameserver 8.8.4.4 \
  internal-subnet-2
  

|---------------|
| Create Router |
|---------------|

openstack router create admin-router-2
openstack router add subnet admin-router-2 internal-subnet-2
openstack router set --external-gateway ext-vlan192-net admin-router-2


|-----------------------|
| Create Security Group |
|-----------------------|

openstack security group create allow-ping-ssh-web

openstack security group rule create --ingress --ethertype IPv4 --protocol icmp allow-ping-ssh-web
openstack security group rule create --ingress --ethertype IPv4 --protocol tcp --dst-port 22 allow-ping-ssh-web
openstack security group rule create --ingress --ethertype IPv4 --protocol tcp --dst-port 80 allow-ping-ssh-web
openstack security group rule create --ingress --ethertype IPv4 --protocol tcp --dst-port 443 allow-ping-ssh-web


|---------------|
| Create Images |
|---------------|

wget http://download.cirros-cloud.net/0.6.1/cirros-0.6.1-x86_64-disk.img

openstack image create \
  --public \
  --disk-format qcow2 \
  --container-format bare \
  --file cirros-0.6.1-x86_64-disk.img \
  cirros-os
  
  
|----------------|
| Create Volumes |
|----------------|

openstack volume create \
  --size 20 \
  --image cirros-os \
  --bootable cirrosvol

openstack volume create --size 60 vol-c
openstack volume create --size 60 vol-d


|------------------|
| Create Instances |
|------------------|

openstack server create \
  --image cirros-os \
  --flavor m1.medium \
  --security-group allow-ping-ssh-web \
  --key-name node01-key \
  --network internal-net-2 \
  instance-smoke-test-1
  
  
|-----------------------|
| Associate Floating IP |
|-----------------------|

openstack floating ip create --floating-ip-address 192.168.192.125 ext-vlan192-net
openstack server add floating ip 192.168.192.125 instance-smoke-test-1
  
  
|-----------------------------------|
| Attach Secondary Disk to Instance |
|-----------------------------------|

openstack server add volume \
  --device /dev/vdb \
  --disable-delete-on-termination \
  instance-smoke-test-1 vol-c
  
  
|---------------------|
| Connect to Instance |
|---------------------|

ping 192.168.192.125
ssh [email protected]