Cloud Storage with Red Hat Ceph Storage

CL260

Welcome

Course Objectives and Structure

Schedule

Chapter 1: Introducing Red Hat Ceph Storage Architecture

Goal: Describe Red Hat Ceph Storage architecture, including data organization, distribution, and client access methods.

Objectives:

• Describe the personas in the cloud storage ecosystem that characterize the use cases and tasks taught in this course.

• Describe the Red Hat Ceph Storage architecture, introduce the Object Storage Cluster, and describe the choices in data access methods.

• Describe and compare the use cases for the various management interfaces provided for Red Hat Ceph Storage.

Quiz: Describing Storage Personas

Guided Exercise: Describing Red Hat Ceph Storage Architecture

Guided Exercise: Describing Red Hat Ceph Storage Management Interfaces

Summary

• The following services provide the foundation for a Ceph storage cluster:
  • Monitors (MONs) maintain cluster maps.
  • Object Storage Devices (OSDs) store and manage objects.
  • Managers (MGRs) track and expose cluster runtime metrics.
  • Metadata Servers (MDSes) store metadata that CephFS uses to efficiently run POSIX commands for clients.
• RADOS (Reliable Autonomic Distributed Object Store) is the back end for storage in the Ceph cluster, a self-healing and self-managing object store.
• RADOS provides four access methods to storage: the librados native API, the object-based RADOS Gateway, the RADOS Block Device (RBD), and the distributed file-based CephFS file system.
• A Placement Group (PG) aggregates a set of objects into a hash bucket. The CRUSH algorithm maps the hash buckets to a set of OSDs for storage.

Summary (continued)

• Pools are logical partitions of the Ceph storage that are used to store object data. Each pool is a name tag for grouping objects. A pool groups objects for storage by using placement groups.
• Red Hat Ceph Storage provides two interfaces, a command line and a Dashboard GUI, for managing clusters. Both interfaces use the same cephadm module to perform operations and to interact with cluster services.

Chapter 2: Deploying Red Hat Ceph Storage

Goal: Deploy a new Red Hat Ceph Storage cluster and expand the cluster capacity.

Objectives:

• Prepare for and perform a Red Hat Ceph Storage cluster deployment using cephadm command-line tools.

• Expand capacity to meet application storage requirements by adding OSDs to an existing cluster.

Guided Exercise: Deploying Red Hat Ceph Storage

Guided Exercise: Expanding Red Hat Ceph Storage Cluster Capacity

Summary

• The two main components of the cephadm utility:
  • The cephadm shell runs a bash shell within a specialized management container. Use the cephadm shell to perform cluster deployment tasks and cluster management tasks after the cluster is installed.
  • The cephadm orchestrator provides a command-line interface to the orchestrator ceph-mgr modules. The orchestrator coordinates configuration changes that must be performed cooperatively across multiple nodes and services in a storage cluster.
• As of version 5.0, all Red Hat Ceph Storage cluster services are containerized.
• Preparing for a new cluster deployment requires planning cluster service placement and distributing SSH keys to nodes.

Summary (continued)

• Use cephadm to bootstrap a new cluster:
  • Installs and starts the MON and MGR daemons on the bootstrap node.
  • Writes a copy of the cluster public SSH key and adds the key to authorized keys file.
  • Writes a minimal configuration file to communicate with the new cluster.
  • Writes a copy of the administrative secret key to the key ring file.
  • Deploys a basic monitoring stack.
• Use the cephadm-preflight.yml playbook to verify cluster host prerequisites.
• Assign labels to the cluster hosts to identify the daemons running on each host. The _admin label is reserved for administrative nodes.
• Expand cluster capacity by adding OSD nodes to the cluster or additional storage space to existing OSD nodes.

Chapter 3: Configuring a Red Hat Ceph Storage Cluster

Goal: Manage the Red Hat Ceph Storage configuration, including the primary settings, the use of monitors, and the cluster network layout.

Objectives:

• Identify and configure the primary settings for the overall Red Hat Ceph Storage cluster.

• Describe the purpose of cluster monitors and the quorum procedures, query the monitor map, manage the configuration database, and describe Cephx.

• Describe the purpose for each of the cluster networks, and view and modify the network configuration.

Guided Exercise: Managing Cluster Configuration Settings

Guided Exercise: Configuring Cluster Monitors

Guided Exercise: Configuring Cluster Networking

Summary

• Most cluster configuration settings are stored in the cluster configuration database on the MON nodes. The database is automatically synchronized across MONs.
• Certain configuration settings, such as cluster boot settings, can be stored in the cluster configuration file. The default file name is ceph.conf. This file must be synchronized manually between all cluster nodes.
• Most configuration settings can be modified when the cluster is running. You can change a setting temporarily or make it persistent across daemon restarts.
• The MON map holds the MON cluster quorum information that can be viewed with ceph commands or with the dashboard. You can configure MON settings to ensure high cluster availability.

Summary (continued)

• Cephx provides cluster authentication via shared secret keys. The client.admin key ring is required for administering the cluster.
• Cluster nodes operate across the public network. You can configure an additional cluster network to separate OSD replication, heartbeat, backfill, and recovery traffic. Cluster performance and security might be increased by configuring a cluster network.
• You can use firewall rules to secure communication to cluster nodes.

Chapter 4: Creating Object Storage Cluster Components

Goal: Create and manage the components that comprise the object storage cluster, including OSDs, pools, and the cluster authorization method.

Objectives:

• Describe OSD configuration scenarios and create BlueStore OSDs using ceph-volume.

• Describe and compare replicated and erasure coded pools, and create and configure each pool type.

• Describe Cephx and configure user authentication and authorization for Ceph clients.

Guided Exercise: Creating BlueStore OSDs Using Logical Volumes

Guided Exercise: Creating and Configuring Pools

Guided Exercise: Managing Ceph Authentication

Summary

• BlueStore is the default storage back end for Red Hat Ceph Storage 5. It stores objects directly on raw block devices and improves performance over the previous FileStore back end.
• BlueStore OSDs use a RocksDB key-value database to manage metadata and store it on a BlueFS partition. Red Hat Ceph Storage 5 uses sharding by default for new OSDs.
• Block.db stores object metadata and the write-ahead log (WAL) stores journals. You can improve OSD performance by placing the block.db and WAL devices on faster storage than the object data.
• You can provision OSDs by using service specification files, by choosing a specific host and device, or automatically with the orchestrator service.
• Pools are logical partitions for storing objects. The available pool types are replicated and erasure coded.
• Replicated pools are the default type of pool, they copy each object to multiple OSDs.

Summary (continued)

• Erasure coded pools function by dividing object data into chunks (k), calculating coding chunks (m) based on the data chunks, then storing each chunk on separate OSDs. The coding chunks are used to reconstruct object data if an OSD fails.
• A pool namespace allows you to logically partition a pool and is useful for restricting storage access by an application.
• The cephx protocol authenticates clients and authorizes communication between clients, applications, and daemons in the cluster. It is based on shared secret keys.
• Clients can access the cluster when they are configured with a user account name and a key-ring file containing the user's secret key.
• Cephx capabilities provide a way to control access to pools and object data within pools.

Chapter 5: Creating and Customizing Storage Maps

Goal: Manage and adjust the CRUSH and OSD maps to optimize data placement to meet the performance and redundancy requirements of cloud applications.

Objectives:

• Administer and update the cluster CRUSH map used by the Ceph cluster.

• Describe the purpose and modification of the OSD maps.

Guided Exercise: Managing and Customizing the CRUSH Map

Guided Exercise: Managing the OSD Map

Summary

• The CRUSH algorithm provides a decentralized way for Ceph clients to interact with the Red Hat Ceph Storage cluster, which enables massive scalability.
• The CRUSH map contains two main components: a hierarchy of buckets that organize OSDs into a treelike structure where the OSDs are the leaves of the tree, and at least one CRUSH rule that determines how Ceph assigns PGs to OSDs from the CRUSH tree.
• Ceph provides various command-line tools to display, tune, modify, and use the CRUSH map.

Summary (continued)

• You can modify the CRUSH algorithm's behavior by using tunables, which disable, enable, or adjust features of the CRUSH algorithm.
• The OSD map epoch is the map's revision number and increments whenever a change occurs. Ceph updates the OSD map every time an OSD joins or leaves the cluster and OSDs keep the map synchronized among themselves.

Chapter 6: Providing Block Storage Using RADOS Block Devices

Goal: Configure Red Hat Ceph Storage to provide block storage for clients using RADOS block devices (RBDs).

Objectives:

• Provide block storage to Ceph clients using RADOS block devices (RBDs), and manage RBDs from the command line.

• Create and configure RADOS block devices snapshots and clones.

• Export an RBD image from the cluster to an external file and import it into another cluster.

Guided Exercise: Managing RADOS Block Devices

Guided Exercise: Managing RADOS Block Device Snapshots

Guided Exercise: Importing and Exporting RBD Images

Summary

• The rbd command manages RADOS block device pools, images, snapshots, and clones.
• The rbd map command uses the krbd kernel module to map RBD images to Linux block devices. Configuring the rbdmap service can map these images persistently.
• RBD has an export and import mechanism for maintaining copies of RBD images that are fully functional and accessible.
• The rbd export-diff and the rbd import-diff commands export and import RBD image changes made between two points in time.

Chapter 7: Expanding Block Storage Operations

Goal: Expand block storage operations by implementing remote mirroring and the iSCSI Gateway.

Objectives:

• Configure an RBD mirror to replicate an RBD block device between two Ceph clusters for disaster recovery purposes.

• Configure the Ceph iSCSI Gateway to export RADOS Block Devices using the iSCSI protocol, and configure clients to use the iSCSI Gateway.

Guided Exercise: Configuring RBD Mirrors

Quiz: Providing iSCSI Block Storage

Summary

• RBD mirroring supports automatic or selective mirroring of images using pool mode or image mode.
• The RBD mirror agent can replicate pool data between two Red Hat Ceph Storage clusters, in either one-way or two-way mode, to facilitate disaster recovery.
• Deploying an iSCSI gateway publishes RBD images as iSCSI targets for network-based block storage provisioning.

Chapter 8: Providing Object Storage Using a RADOS Gateway

Goal: Configure Red Hat Ceph Storage to provide object storage for clients using a RADOS Gateway (RGW).

Objectives:

• Deploy a RADOS Gateway to provide clients with access to Ceph object storage.

• Configure the RADOS Gateway with multisite support to allow objects to be stored in two or more geographically diverse Ceph storage clusters.

Guided Exercise: Deploying an Object Storage Gateway

Guided Exercise: Configuring a Multisite Object Storage Deployment

Summary

• The RADOS Gateway is a service that connects to a Red Hat Ceph Storage cluster, and provides object storage to applications using a REST API.
• You can deploy the RADOS Gateway by using the Ceph orchestrator command-line interface or by using a service specification file.
• You can use HAProxy and keepalived services to load balance the RADOS Gateway service.

Summary (continued)

• The RADOS Gateway supports multisite configuration, which allows RADOS Gateway objects to be replicated between separate Red Hat Ceph Storage clusters.
• Objects written to a RADOS Gateway for one zone are replicated to all other zones in the zone group.
• Metadata and configuration updates must occur in the master zone of the master zone group.

Chapter 9: Accessing Object Storage Using a REST API

Goal: Configure the RADOS Gateway to provide access to object storage using REST APIs.

Objectives:

• Configure the RADOS Gateway to provide access to object storage compatible with the Amazon S3 API, and manage objects stored using that API.

• Configure the RADOS Gateway to provide access to object storage compatible with the Swift API, and manage objects stored using that API.

Guided Exercise: Providing Object Storage Using the Amazon S3 API

Guided Exercise: Providing Object Storage Using the Swift API

Summary

• You can access the RADOS Gateway by using clients that are compatible with the Amazon S3 API or the OpenStack Swift API.
• The RADOS Gateway can be configured to use either of the bucket name formats supported by the Amazon S3 API.
• To support authentication by using the OpenStack Swift API, Swift users are represented by RADOS Gateway subusers.
• You can define deletion policies for Amazon S3 buckets, and object versioning for Swift containers, to manage the behavior of deleted objects.

Chapter 10: Providing File Storage with CephFS

Goal: Configure Red Hat Ceph Storage to provide file storage for clients using the Ceph File System (CephFS).

Objectives:

• Provide file storage on the Ceph cluster by deploying the Ceph File System (CephFS).

• Configure CephFS, including snapshots, replication, memory management, and client access.

Guided Exercise: Deploying Shared File Storage

Guided Exercise: Managing Shared File Storage

Summary

• You can distinguish the different characteristics for file-based, block-based, and object-based storage.
• CephFS is a POSIX-compliant file system that is built on top of RADOS to provide file-based storage.
• CephFS requires at least one Metadata Server that is separate from file data.
• Deploying CephFS requires multiple steps:
  • Create two pools, one for CephFS data and another for CephFS metadata.
  • Start the MDS service on the hosts.
  • Create a CephFS file system.
• You can mount CephFS file systems with either of the two available clients:
  • The kernel client, which does not support quotas but is faster.
  • The FUSE client, which supports quotas as ACLs but is slower.

Summary (continued)

• NFS Ganesha is a user space NFS file server for accessing Ceph storage.
• CephFS supports multisite geo-replication with snapshots.
• You can determine which OSDs store a file's objects.
• You can modify the RADOS layout to control how files are mapped to objects.
• CephFS enables asynchronous snapshots by creating a folder in the hidden .snap folder.
• You can schedule snapshots for your CephFS file system.

Chapter 11: Managing a Red Hat Ceph Storage Cluster

Goal: Manage an operational Ceph cluster using tools to check status, monitor services, and properly start and stop all or part of the cluster. Perform cluster maintenance by replacing or repairing cluster components, including MONs, OSDs, and PGs.

Objectives:

• Administer and monitor a Red Hat Ceph Storage cluster, including starting and stopping specific services or the full cluster, and querying cluster health and utilization.

• Perform common cluster maintenance tasks, such as adding or removing MONs and OSDs, and recovering from various component failures.

Guided Exercise: Performing Cluster Administration and Monitoring

Guided Exercise: Performing Cluster Maintenance Operations

Summary

• Enable or disable Ceph Manager (MGR) modules, and more about the role of the Ceph Manager (MGR).
• Use the CLI to find the URL of the Dashboard GUI on the active MGR.
• View the status of cluster MONs by using the CLI or the Dashboard GUI.
• Monitor cluster health and interpret the cluster health status.
• Power down the entire cluster by setting cluster flags to stop background operations, then stopping daemons and nodes in a specific order by function.
• Power up the entire cluster by starting nodes and daemons in a specific order by function, then setting cluster flags to enable background operations.

Summary (continued)

• Start, stop, or restart individual cluster daemons and view daemon logs.
• Monitor cluster storage by viewing OSD and PG states and capacity.
• Identify the PG details for a specific object.
• Find and replace a failed OSD.
• Add or remove a cluster MON node.
• Use the balancer module to optimize the placement of PGs across OSDs.

Chapter 12: Tuning and Troubleshooting Red Hat Ceph Storage

Goal: Identify the key Ceph cluster performance metrics, and use them to tune and troubleshoot Ceph operations for optimal performance.

Objectives:

• Choose Red Hat Ceph Storage architecture scenarios and operate Red Hat Ceph Storage-specific performance analysis tools to optimize cluster deployments.

• Protect OSD and cluster hardware resources from over-utilization by controlling scrubbing, deep scrubbing, backfill, and recovery processes to balance CPU, RAM, and I/O requirements.

• Identify key tuning parameters and troubleshoot performance for Ceph clients, including RADOS Gateway, RADOS Block Devices, and CephFS.

Guided Exercise: Optimizing Red Hat Ceph Storage Performance

Guided Exercise: Tuning Object Storage Cluster Performance

Guided Exercise: Troubleshooting Clusters and Clients

Summary

• Red Hat Ceph Storage 5 performance depends on the performance of the underlying storage, network, and operating system file system components.
• Performance is improved by reducing latency, increasing IOPS, and increasing throughput. Tuning for one metric often adversely affects the performance of another. Your primary tuning metric must consider the expected workload behavior of your storage cluster.
• Ceph implements a scale-out model architecture. Increasing the number of OSD nodes increases the overall performance. The greater the parallel access, the greater the load capacity.
• The RADOS and RBD bench commands are used to stress and benchmark a Ceph cluster.

Summary (continued)

• Controlling scrubbing, deep scrubbing, backfill, and recovery processes helps avoid cluster over-utilization.
• Troubleshooting Ceph issues starts with determining which Ceph component is causing the issue.
• Enabling logging for a failing Ceph subsystem provides diagnostic information about the issue.
• The log debug level is used to increase the logging verbosity.

Chapter 13: Managing Cloud Platforms with Red Hat Ceph Storage

Goal: Manage Red Hat cloud infrastructure to use Red Hat Ceph Storage to provide image, block, volume, object, and shared file storage.

Objectives:

• Describe Red Hat OpenStack Platform storage requirements, and compare the architecture choices for using Red Hat Ceph Storage as an RHOSP storage back end.

• Describe how OpenStack implements Ceph storage for each storage-related OpenStack component.

• Describe Red Hat OpenShift Container Platform storage requirements, and compare the architecture choices for using Red Hat Ceph Storage as an RHOCP storage back end.

• Describe how OpenShift implements Ceph storage for each storage-related OpenShift feature.

Quiz: Introducing OpenStack Storage Architecture

Quiz: Implementing Storage in OpenStack Components

Quiz: Introducing OpenShift Storage Architecture

Quiz: Implementing Storage in OpenShift Components

Summary

• Red Hat Ceph Storage can provide a unified storage back end for OpenStack services that consume block, image, object, and file-based storage.
• OpenStack Glance can use Ceph RBD images to store the operating system images that it manages.
• OpenStack Cinder can also use RADOS block devices to provide block-based storage for virtual machines that run as cloud instances.
• The RADOS Gateway can replace the native OpenStack Swift storage by providing object storage for applications that use the OpenStack Swift API, and integrates its user authentication with OpenStack Keystone.
• Red Hat OpenShift Data Foundation is an operator bundle that provides cloud storage and data services to Red Hat OpenShift Container Platform; it is composed of the ocs-storage, NooBaa, and Rook-Ceph operators.

Summary (continued)

• Rook-Ceph is a cloud storage orchestrator that installs, monitors, and manages the underlying Ceph cluster in the OpenShift Data Foundation bundle operator. Rook-Ceph provides the required drivers to request storage to the cluster.
• PersistentVolumeClaims are an OpenShift resource type that represent a request for a storage object. They contain the StorageClass which describes the PersistentVolume that should bind to it.
• Access modes describe the mount capabilities of a PersistentVolume on pods.

Chapter 14: Comprehensive Review

Goal: Review tasks from Cloud Storage with Red Hat Ceph Storage

Objectives:

• Review tasks from Cloud Storage with Red Hat Ceph Storage

Lab: Deploying Red Hat Ceph Storage

Lab: Configuring Red Hat Ceph Storage

Lab: Deploying CephFS

Lab: Deploying and Configuring Block Storage with RBD

Lab: Deploying and Configuring RADOS Gateway