Keith Tenzer, Red Hat solutions architect, discusses why OpenStack and Ceph are an amazing team.

In this article, we’ll discuss why Ceph is perfect fit for OpenStack.

We’ll also show you how to integrate three prominent OpenStack use cases with Ceph: Cinder (block storage), Glance (images) and Nova (VM virtual disks).

Ceph provides unified scale-out storage, using commodity x86 hardware that is self-healing and intelligently anticipates failures. It has become the de facto standard for software-defined storage. Because Ceph is open source, it has enabled many vendors the ability to provide Ceph based software-defined storage systems. Ceph is not just limited to companies like Red Hat, Suse, Mirantis, Ubuntu, etc. — SanDisk, Fujitsu, HP, Dell, Samsung and many more now offer integrated solutions, too. There are even large-scale community built environments — CERN comes to mind — that provide storage services for 10,000s of VMs.

Ceph is by no means limited to OpenStack, however this is where Ceph started gaining traction. Looking at latest OpenStack user survey, Ceph is the clear leader for OpenStack storage. Page 42 in OpenStack April 2016 User Survey reveals Ceph is 57 percent of OpenStack storage. The next is LVM (local storage) with 28 percent followed by NetApp with 9 percent. If we remove LVM, Ceph leads any other storage company by 48 percent, that is incredible. Why is that?

There are several reasons, but here are my top three:

  • Ceph is a scale-out unified storage platform. OpenStack needs two things from storage: ability to scale with OpenStack itself and do so regardless of block (Cinder), File (Manila) or Object (Swift). Traditional storage vendors need to provide two or three different storage systems to achieve this. They don’t scale the same and in most cases only scale-up in never-ending migration cycles. Their management capabilities are never truly integrated across broad spectrum of storage use-cases.
  • Ceph is cost-effective. Ceph leverages Linux as an operating system instead of something proprietary. You can choose not only whom you purchase Ceph from, but also where you get your hardware. It can be same vendor or different. You can purchase commodity hardware, or even buy integrated solution of Ceph + Hardware from single vendor. There are even hyper-converged options for Ceph that are emerging (running Ceph services on compute nodes).
  • Ceph is OpenSource project just like OpenStack. This allows for a much tighter integration and cross-project development. Proprietary vendors are always playing catch-up since they have secrets to protect and their influence is usually limited in Opensource communities, especially in OpenStack context.

Here is an architecture diagram that shows all the different OpenStack components that need storage. It shows how they integrate with Ceph and how Ceph provides a unified storage system that scales to fill all these use cases.

If you are interested in more topics relating to Ceph and OpenStack, I recommend this.

Ok enough talking about why Ceph and OpenStack are so great, let’s get our hands dirty and see how to hook it up!

If you don’t have a Ceph environment you can follow this article on how to set one up quickly.

Glance Integration

Glance is the image service within OpenStack. By default images are stored locally on controllers and then copied to compute hosts when requested. The compute hosts cache the images but they need to be copied again, every time an image is updated.

Ceph provides back end for Glance allowing images to be stored in Ceph, instead of locally on controller and compute nodes. This greatly reduces network traffic for pulling images and increases performance since Ceph can clone images instead of copying them. In addition, it makes migrating between OpenStack deployments or concepts like multi-site OpenStack much simpler.

Install the Ceph client used by Glance.

    [[email protected] ~]# yum install -y python-rbd

Create Ceph user and set home directory to /etc/ceph.

    [[email protected] ~]# mkdir /etc/ceph
    [[email protected] ~]# useradd ceph
    [[email protected] ~]# passwd ceph

Add Ceph user to sudoers.

    cat << EOF >/etc/sudoers.d/ceph
    ceph ALL = (root) NOPASSWD:ALL
    Defaults:ceph !requiretty

On Ceph admin node.
Create Ceph RBD Pool for Glance images.

  [[email protected] ~]$ sudo ceph osd pool create images 128

Create keyring that will allow Glance access to pool.

  [[email protected] ~]$ sudo ceph auth get-or-create client.images mon 'allow r' osd 'allow class-read object_prefix rdb_children, allow rwx pool=images' -o /etc/ceph/ceph.client.images.keyring

Copy the keyring to /etc/ceph on OpenStack controller.

  [[email protected] ~]$ scp /etc/ceph/ceph.conf [email protected]:/etc/ceph

Set permissions so Glance can access Ceph keyring.

  [[email protected] ~(keystone_admin)]# chgrp glance /etc/ceph/ceph.client.images.keyring
[[email protected] ~(keystone_admin)]#chmod 0640 /etc/ceph/ceph.client.images.keyring

Add keyring file to Ceph configuration.

  [[email protected] ~(keystone_admin)]# vi /etc/ceph/ceph.conf
keyring = /etc/ceph/ceph.client.images.keyring

Create a backup of original Glance configuration.

  [[email protected] ~(keystone_admin)]# cp /etc/glance/glance-api.conf /etc/glance/glance-api.conf.orig

Update Glace configuration

  [[email protected] ~]# vi /etc/glance/glance-api.conf
  stores =
  default_store = rbd
  rbd_store_pool = images
  rbd_store_user = images
  rbd_store_ceph_conf = /etc/ceph/ceph.conf

Restart Glance.

  [[email protected] ~(keystone_admin)]# systemctl restart openstack-glance-api

Download Cirros images and add it into Glance.

  [[email protected] ~(keystone_admin)]# wget

Convert QCOW2 to RAW. It is recommended for Ceph to always use RAW format.

  [[email protected] ~(keystone_admin)]#qemu-img convert cirros-0.3.4-x86_64-disk.img cirros-0.3.4-x86_64-disk.raw

Add image to Glance.

  [[email protected] ~(keystone_admin)]#glance image-create --name "Cirros 0.3.4" --disk-format raw --container-format bare --visibility public --file cirros-0.3.4-x86_64-disk.raw

| Property | Value |
| checksum | ee1eca47dc88f4879d8a229cc70a07c6 |
| container_format | bare |
| created_at | 2016-09-07T12:29:23Z |
| disk_format | qcow2 |
| id | a55e9417-67af-43c5-a342-85d2c4c483f7 |
| min_disk | 0 |
| min_ram | 0 |
| name | Cirros 0.3.4 |
| owner | dd6a4ed994d04255a451da66b68a8057 |
| protected | False |
| size | 13287936 |
| status | active |
| tags | [] |
| updated_at | 2016-09-07T12:29:27Z |
| virtual_size | None |
| visibility | public |

Check that a Glance image exists in Ceph.

  [[email protected] ceph-config]$ sudo rbd ls images
  [[email protected] ceph-config]$ sudo rbd info images/a55e9417-67af-43c5-a342-85d2c4c483f7
rbd image 'a55e9417-67af-43c5-a342-85d2c4c483f7':
 size 12976 kB in 2 objects
 order 23 (8192 kB objects)
 block_name_prefix: rbd_data.183e54fd29b46
 format: 2
 features: layering, striping
 stripe unit: 8192 kB
 stripe count: 1

Cinder Integration

Cinder is the block storage service in OpenStack. Cinder provides an abstraction around block storage and allows vendors to integrate by providing a driver. In Ceph, each storage pool can be mapped to a different Cinder back end. This allows for creating storage services such as gold, silver or bronze. You can decide for example that gold should be fast SSD disks that are replicated three times, while silver only should be replicated two times and bronze should use slower disks with erasure coding.

Create a Ceph pool for cinder volumes.

  [[email protected] ~]$ sudo ceph osd pool create  128

Create a keyring to grant Cinder access.

  [[email protected] ~]$ sudo ceph auth get-or-create client.volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images' -o /etc/ceph/ceph.client.volumes.keyring

Copy the keyring to OpenStack controllers.

  [[email protected] ~]$ scp /etc/ceph/ceph.client.volumes.keyring [email protected]:/etc/ceph

Create a file that contains just the authentication key on OpenStack controllers.

  [[email protected] ~]$ sudo ceph auth get-key client.volumes |ssh osp9.lab tee client.volumes.key

Set permissions on the keyring file so it can be accessed by Cinder.

  [[email protected] ~(keystone_admin)]# chgrp cinder /etc/ceph/ceph.client.volumes.keyring
[[email protected] ~(keystone_admin)]# chmod 0640 /etc/ceph/ceph.client.volumes.keyring

Add the keyring to Ceph configuration file on OpenStack controllers.

  [[email protected] ~(keystone_admin)]#vi /etc/ceph/ceph.conf

keyring = /etc/ceph/ceph.client.volumes.keyring

Give KVM Hypervisor access to Ceph.

  [[email protected] ~(keystone_admin)]# uuidgen |tee /etc/ceph/cinder.uuid.txt

Create a secret in virsh so KVM can access Ceph pool for cinder volumes.

 [[email protected] ~(keystone_admin)]#vi /etc/ceph/cinder.xml

 client.volumes secret

  [[email protected] ceph]# virsh secret-define --file /etc/ceph/cinder.xml

  [[email protected] ~(keystone_admin)]# virsh secret-set-value --secret ce6d1549-4d63-476b-afb6-88f0b196414f --base64 $(cat /etc/ceph/client.volumes.key)

Add a Ceph backend for Cinder

[[email protected] ~(keystone_admin)]#vi /etc/cinder/cinder.conf

volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rbd_store_chunk_size = 4
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = volumes
rbd_secret_uuid = ce6d1549-4d63-476b-afb6-88f0b196414f

Restart Cinder service on all controllers.

  [[email protected] ~(keystone_admin)]# openstack-service restart cinder

Create a Cinder volume.

  [[email protected] ~(keystone_admin)]# cinder create --display-name="test" 1
  | Property | Value |
  | attachments | [] |
  | availability_zone | nova |
  | bootable | false |
  | consistencygroup_id | None |
  | created_at | 2016-09-08T10:58:17.000000 |
  | description | None |
  | encrypted | False |
  | id | d251bb74-5c5c-4c40-a15b-2a4a17bbed8b |
  | metadata | {} |
  | migration_status | None |
  | multiattach | False |
  | name | test |
  | os-vol-host-attr:host | None |
  | os-vol-mig-status-attr:migstat | None |
  | os-vol-mig-status-attr:name_id | None |
  | os-vol-tenant-attr:tenant_id | dd6a4ed994d04255a451da66b68a8057 |
  | replication_status | disabled |
  | size | 1 |
  | snapshot_id | None |
  | source_volid | None |
  | status | creating |
  | updated_at | None |
  | user_id | 783d6e51e611400c80458de5d735897e |
  | volume_type | None |
  List new cinder volume
  [[email protected] ~(keystone_admin)]# cinder list
  | ID | Status | Name | Size | Volume Type | Bootable | Attached to |
  | d251bb74-5c5c-4c40-a15b-2a4a17bbed8b | available | test | 1 | - | false | |

List a Cinder volume in Ceph.

  [[email protected] ~]$ sudo rbd ls volumes
   [[email protected] ~]$ sudo rbd info volumes/volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b
  rbd image 'volume-d251bb74-5c5c-4c40-a15b-2a4a17bbed8b':
   size 1024 MB in 256 objects
   order 22 (4096 kB objects)
   block_name_prefix: rbd_data.2033b50c26d41
   format: 2
   features: layering, striping
   stripe unit: 4096 kB
   stripe count: 1

Integrating Ceph with Nova Compute

Nova is the compute service within OpenStack. Nova stores virtual disks images associated with running VMs by default, locally on Hypervisor under /var/lib/nova/instances. There are a few drawbacks to using local storage on compute nodes for virtual disk images:

  • Images are stored under root filesystem. Large images can cause filesystem to fill up, thus crashing compute nodes.
  • A disk crash on compute node could cause loss of virtual disk and as such a VM recovery would be impossible.

Ceph is one of the storage backends that can integrate directly with Nova. In this section we will see how to configure that.

  [[email protected] ~]$ sudo ceph osd pool create vms 128

Create an authentication keyring for Nova.

  [[email protected] ~]$ sudo ceph auth get-or-create client.nova mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=vms, allow rx pool=images' -o /etc/ceph/ceph.client.nova.keyring

Copy the keyring to OpenStack controllers.

  [[email protected] ~]$ scp /etc/ceph/ceph.client.nova.keyring [email protected]:/etc/ceph

Create key file on OpenStack controllers.

  [[email protected] ~]$ sudo ceph auth get-key client.nova |ssh osp9.lab tee client.nova.key

Set permissions on the keyring file so it can be accessed by Nova service.

  [[email protected] ~(keystone_admin)]# chgrp nova /etc/ceph/ceph.client.nova.keyring
[[email protected] ~(keystone_admin)]# chmod 0640 /etc/ceph/ceph.client.nova.keyring

Ensure the required rpm packages are installed.

  [[email protected] ~(keystone_admin)]# yum list installed python-rbd ceph-common
  Loaded plugins: product-id, search-disabled-repos, subscription-manager
  Installed Packages
  ceph-common.x86_64 1:0.94.5-15.el7cp @rhel-7-server-rhceph-1.3-mon-rpms
  python-rbd.x86_64 1:0.94.5-15.el7cp @rhel-7-server-rhceph-1.3-mon-rpms

Update the Ceph configuration.

  [[email protected] ~(keystone_admin)]#vi /etc/ceph/ceph.conf

keyring = /etc/ceph/ceph.client.nova.keyring

Give KVM access to Ceph.

  [[email protected] ~(keystone_admin)]# uuidgen |tee /etc/ceph/nova.uuid.txt

Create a secret in virsh so KVM can access the Ceph pool for cinder volumes.

  [[email protected] ~(keystone_admin)]#vi /etc/ceph/nova.xml
  client.volumes secret
  [[email protected] ~(keystone_admin)]# virsh secret-define --file /etc/ceph/nova.xml
  [[email protected] ~(keystone_admin)]# virsh secret-set-value --secret c89c0a90-9648-49eb-b443-c97adb538f23 --base64 $(cat /etc/ceph/client.nova.key)

Make a backup of Nova configuration

  [[email protected] ~(keystone_admin)]# cp /etc/nova/nova.conf /etc/nova/nova.conf.orig

Update the Nova configuration to use Ceph back end.

  [[email protected] ~(keystone_admin)]#vi /etc/nova/nova.conf
force_raw_images = True
disk_cachemodes = writeback

images_type = rbd
images_rbd_pool = vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_user = nova
rbd_secret_uuid = c89c0a90-9648-49eb-b443-c97adb538f23

Restart Nova services.

  [[email protected] ~(keystone_admin)]# systemctl restart openstack-nova-compute

List Neutron networks.

  [[email protected] ~(keystone_admin)]# neutron net-list
| id | name | subnets |
| 4683d03d-30fc-4dd1-9b5f-eccd87340e70 | private | ef909061-34ee-4b67-80a9-829ea0a862d0 |
| 8d35a938-5e4f-46a2-8948-b8c4d752e81e | public | bb2b65e7-ab41-4792-8591-91507784b8d8 |

Start an ephemeral VM instance using Cirros image that was added in the steps for Glance.

  [[email protected] ~(keystone_admin)]# nova boot --flavor m1.small --nic net-id=4683d03d-30fc-4dd1-9b5f-eccd87340e70 --image='Cirros 0.3.4' cephvm
| Property | Value |
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | - |
| OS-EXT-SRV-ATTR:hypervisor_hostname | - |
| OS-EXT-SRV-ATTR:instance_name | instance-00000001 |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | wzKrvK3miVJ3 |
| config_drive | |
| created | 2016-09-08T11:41:29Z |
| flavor | m1.small (2) |
| hostId | |
| id | 85c66004-e8c6-497e-b5d3-b949a1666c90 |
| image | Cirros 0.3.4 (a55e9417-67af-43c5-a342-85d2c4c483f7) |
| key_name | - |
| metadata | {} |
| name | cephvm |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | dd6a4ed994d04255a451da66b68a8057 |
| updated | 2016-09-08T11:41:33Z |
| user_id | 783d6e51e611400c80458de5d735897e |

Wait until the VM is active.

  [[email protected] ceph(keystone_admin)]# nova list
| ID | Name | Status | Task State | Power State | Networks |
| 8ca3e74e-cd52-42a6-acec-13a5b8bda53c | cephvm | ACTIVE | - | Running | private= |

List images in Ceph vms pool. We should now see the image is stored in Ceph.

  [[email protected] ~]$ sudo rbd -p vms ls



  [[email protected] ceph(keystone_admin)]# nova image-list
| ID | Name | Status | Server |
| a55e9417-67af-43c5-a342-85d2c4c483f7 | Cirros 0.3.4 | ACTIVE | |
| 34510bb3-da95-4cb1-8a66-59f572ec0a5d | test123 | ACTIVE | |
| cf56345e-1454-4775-84f6-781912ce242b | test456 | ACTIVE | |
  [[email protected] ceph(keystone_admin)]# rbd -p images snap unprotect [email protected]
[[email protected] ceph(keystone_admin)]# rbd -p images snap rm [email protected]
[[email protected] ceph(keystone_admin)]# glance image-delete cf56345e-1454-4775-84f6-781912ce242b


In this article, we discussed how OpenStack and Ceph fit perfectly together. We discussed some of the use cases around Glance, Cinder and Nova. Finally we went through steps to integrate Ceph with those use cases. I hope you enjoyed the article and found the information useful. Please share your feedback.


Happy Cephing!

This post first appeared on Keith Tenzer’s blog. Superuser is always interested in community content, email: [email protected]

Cover Photo // CC BY NC

  • Ricardo Bruel

    Thanks for the article helped me a lot.

    There are adicional configurations needed to ensure the quick download of the images.

  • Amjad Yaseen


    I have followed what you did for nova integration with openstack neuton, but am facing issues with no suitable host errors if I used rbd_user = nova, while use cinder user its working fine but no disk under vms pool.

    What the root cause for this.