介绍

手册：https://docs.ceph.com/en/latest/

ceph可以实现的存储方式：

块存储：提供像普通硬盘一样的存储，为使用者提供“硬盘”

文件系统存储：类似于NFS的共享方式，为使用者提供共享文件夹

对象存储：像百度云盘一样，需要使用单独的客户端

ceph还是一个分布式的存储系统，非常灵活。如果需要扩容，只要向ceph集中增加服务器即可。ceph存储数据时采用多副本的方式进行存储，生产环境下，一个文件至少要存3份。ceph默认也是三副本存储。

ceph的构成

Ceph OSD 守护进程： Ceph OSD 用于存储数据。此外，Ceph OSD 利用 Ceph 节点的 CPU、内存和网络来执行数据复制、纠删代码、重新平衡、恢复、监控和报告功能。存储节点有几块硬盘用于存储，该节点就会有几个osd进程。

Ceph Mon监控器： Ceph Mon维护 Ceph 存储集群映射的主副本和 Ceph 存储群集的当前状态。监控器需要高度一致性，确保对Ceph 存储集群状态达成一致。维护着展示集群状态的各种图表，包括监视器图、 OSD 图、归置组（ PG ）图、和 CRUSH 图。

MDSs: Ceph 元数据服务器（ MDS ）为 Ceph 文件系统存储元数据。

RGW：对象存储网关。主要为访问ceph的软件提供API接口。

ceph集群安装

1、环境：

操作系统： Rocky8.4

主机名	IP	角色
ceph-admin	192.168.50.66	cephadm,mon,mgr,osd,rgw
ceph-node01	192.168.50.66	mon,mgr,osd,rgw
ceph-node02	192.168.50.66	mon,mgr,rosd,rgw

2、ceph版本

15.2.17 octopus (stable)

3、初始化工作(三台机器同时操作)：

3.1、关闭防火墙：

systemctl stop firewalld && systemctl disable firewalld

3.1关闭防火墙：

setenforce 0 && sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
[root@ceph-admin ~]# cat /etc/selinux/config
 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

3.3 设置时间同步（很重要，不成功会影响存储创建）：

[root@ceph-admin ~]# cat /etc/chrony.conf
# 使用公共 NTP 服务器
server 0.pool.ntp.org iburst
server 1.pool.ntp.org iburst
server 2.pool.ntp.org iburst
server 3.pool.ntp.org iburst

# 允许本地网络内的客户端同步时间
allow 192.168.50.0/24

# 其他配置项
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chrony


[root@ceph-node01 ~]# cat /etc/chrony.conf
# 使用公共 NTP 服务器
server 192.168.50.66 iburst

# 允许本地网络内的客户端同步时间
allow 192.168.50.0/24

# 其他配置项
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chrony


[root@ceph-node02 ~]# cat /etc/chrony.conf
# 使用公共 NTP 服务器
server 192.168.50.66 iburst


# 允许本地网络内的客户端同步时间
allow 192.168.50.0/24

# 其他配置项
driftfile /var/lib/chrony/drift
makestep 1.0 3
rtcsync
keyfile /etc/chrony.keys
leapsectz right/UTC
logdir /var/log/chrony


#重启时间同步服务
systemctl restart chronyd
ystemctl enable chronyd

3.4 增加ceph安装所需要的yum源

ceph源

[root@ceph-admin ~]# cat /etc/yum.repos.d/ceph.repo
[ceph]
name=Ceph packages for $basearch
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-15.2.17/el8/x86_64
enabled=1
gpgcheck=0


[ceph-noarch]
name=Ceph noarch packages
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-15.2.17/el8/noarch/
enabled=1
gpgcheck=0

[ceph-source]
name=Ceph source packages
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-15.2.17/el8/SRPMS
enabled=1
gpgcheck=0

epel源（不装的话会导致安装不了ceph）

dnf install epel-release -y

3.5 设置主机名：

hostnamectl set-hostname ceph-admin
hostnamectl set-hostname ceph-node01
hostnamectl set-hostname ceph-node02

3.6 修改域名解析文件：


[root@ceph-admin ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.50.66 ceph-admin s3.myorg
192.168.50.67 ceph-node01
192.168.50.68 ceph-node02

3.7 安装ceph所需要的依赖

yum install -y python3 ceph-common device-mapper-persistent-data lvm2

3.8安装docker(三台机器同时操作)：(Rokcy8中默认有podman)

yum install -y yum-utils
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install docker-ce-3:19.03.15 docker-ce-cli-1:19.03.15-3.el8 containerd.io -y
mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
	"registry-mirrors": ["https://fxt824bw.mirror.aliyuncs.com"]
}
EOF


#验证
[root@ceph-admin ~]# docker -v   
Docker version 19.03.15, build 99e3ed8919

#启动docker
systemctl start docker && systemctl enable docker

4、安装cephadm(ceph-admin节点操作)部署集群：

4.1安装cephadm：

yum install cephadm -y

$ cephadm bootstrap --mon-ip ceph-admin的IP
此操作比较慢，要从镜像源拉取镜像
 
# 执行结果：
Verifying podman|docker is present...
Verifying lvm2 is present...
Verifying time synchronization is in place...
Unit chronyd.service is enabled and running
Repeating the final host check...
podman|docker (/usr/bin/docker) is present
systemctl is present
lvcreate is present
Unit chronyd.service is enabled and running
Host looks OK
Cluster fsid: 09feacf4-4f13-11ed-a401-000c29d6f8f4
Verifying IP 192.168.50.66 port 3300 ...
Verifying IP 192.168.50.66 port 6789 ...
Mon IP 192.168.50.66 is in CIDR network 192.168.50.0/24
- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network
Pulling container image docker.io/ceph/ceph:v16...
Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
Extracting ceph user uid/gid from container image...
Creating initial keys...
Creating initial monmap...
Creating mon...
Waiting for mon to start...
Waiting for mon...
mon is available
Assimilating anything we can from ceph.conf...
Generating new minimal ceph.conf...
Restarting the monitor...
Setting mon public_network to 192.168.150.0/24
Wrote config to /etc/ceph/ceph.conf
Wrote keyring to /etc/ceph/ceph.client.admin.keyring
Creating mgr...
Verifying port 9283 ...
Waiting for mgr to start...
Waiting for mgr...
mgr not available, waiting (1/15)...
mgr not available, waiting (2/15)...
mgr not available, waiting (3/15)...
mgr not available, waiting (4/15)...
mgr is available
Enabling cephadm module...
Waiting for the mgr to restart...
Waiting for mgr epoch 5...
mgr epoch 5 is available
Setting orchestrator backend to cephadm...
Generating ssh key...
Wrote public SSH key to /etc/ceph/ceph.pub
Adding key to root@localhost authorized_keys...
Adding host ceph1...
Deploying mon service with default placement...
Deploying mgr service with default placement...
Deploying crash service with default placement...
Enabling mgr prometheus module...
Deploying prometheus service with default placement...
Deploying grafana service with default placement...
Deploying node-exporter service with default placement...
Deploying alertmanager service with default placement...
Enabling the dashboard module...
Waiting for the mgr to restart...
Waiting for mgr epoch 13...
mgr epoch 13 is available
Generating a dashboard self-signed certificate...
Creating initial admin user...
Fetching dashboard port number...
Ceph Dashboard is now available at:
 
             URL: https://ceph1:8443/
            User: admin
        Password: mo5ahyp1wx
 
You can access the Ceph CLI with:
 
        sudo /usr/sbin/cephadm shell --fsid 09feacf4-4f13-11ed-a401-000c29d6f8f4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
 
Please consider enabling telemetry to help improve Ceph:
 
        ceph telemetry on
 
For more information see:
 
        https://docs.ceph.com/docs/pacific/mgr/telemetry/
 
Bootstrap complete.

cephadm v15.2.17 安装部署 ceph 集群

介绍手册：https://docs.ceph.com/en/latest/ ceph可以实现的存储方式：块存储：提供像普通硬盘一样的存储，为使用者提供“硬盘”

介绍