分布式存储Ceph解决方案
合集下载
相关主题
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
Agenda 1 2 3 4
Ceph基础 Ceph 优化 稳定 可靠性 Ceph和OpenStack整合 Ceph运维
2
Ceph定义
Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability Ceph uniquely delivers object, block, and file storage in one unified system
1.ceph osd pool create test 1024 1024 2.rbd -p test create myimage --size 10240 3.rbd map myimage -p test 4.mkfs.xfs -f /dev/rbd/test/myimage 5.mkdir /mnt/test 6.mount rw, /dev/rbd/test/myimage /mnt/test
I/O调度
SSD
网络
增加jumbo值 万兆网卡/交换机 网络隔离 Bond:load Infiniband
NOOP，Ohters Deadline
echo deadline > /sys/block/sdX/queue/scheduler
增大请求队列长度
echo 4096 > /sys/block/sdX/queue/nr_requests
I/O 或Buffered I/O
文件系统
XFS 4k
对齐 GPT parted mount 参数 rw,noatime,nodiratime,nobarrier,inode64,l ogbufs=8,logbsize=256k,delaylog
QEMU
Virtio
scsi 支持多队列 cache rbd cache
Controlled, Scalable, Decentralized Placement of Replicated Data
条带化
stripe_unit = 1048576 stripe_count = 4 Order = 22 size = 4M* 8 = 32M
Erasure Coding
块存储 Qemu RBD
<disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none'/> <auth username='compute'> <secret type='ceph' uuid='a5d0dd94-57c4-ae55-ffe0-7e3732a24455'/> </auth> <source protocol='rbd' name='test/myimage'> <host name='192.168.0.2' port='6789'/> <host name='192.168.0.3' port='6789'/> <host name='192.168.0.6' port='6789'/> </source> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk>
写流程
Rebalance
Cache Pool Tiering
文件存储
文件存储
文件存储
对象存储
对象的写入、读取、复制、更新、删除、访问许可设置、 元数据读取或更新等 REST API, Put ,Get, Post, Delete S3 and Swift API
块存储 linux Kernel RBD
Snapshot
Ceph逐层优化
Qemu RBD Linux Kernel RBD
文件读/写命令 文件读/写命令 文件读/写命令 文件系统 文件系统 文件系统 page cache page cache I/O调度 page cache I/O调度 block device TCP block device QEMU Librbd TCP I/O调度 block device Disk Controller Disk I/O调度 block device Disk Controller Disk page cache 文件系统 文件读/写命令
OSD NODE
Ceph OSD
OSD NODE
Ceph OSD
优化参数
客户端读/写程序
选择使用系统调用或c接口
libaio+Direct 关闭页交换
page cache
vm.swappiness = 0 调整缓存大小
$ echo 5 > /proc/sys/vm/dirty_background_ratio $ echo 10 > /proc/sys/Sage Weil
started the open source Ceph project in 2004 for his doctoral dissertation at the University of California, Santa Cruz
• Inktank
Inktank was a professional services and support company for the open source Ceph file system in 2012
• RedHat
30 April 2014,Red Hat acquire Inktank
Storage for $175 Million
Ceph结构
Ceph代码调用栈
RADOS (Reliable, Autonomic Distributed Object Store)
Ceph网络
文件到osd映射
Ceph基础 Ceph 优化 稳定 可靠性 Ceph和OpenStack整合 Ceph运维
2
Ceph定义
Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability Ceph uniquely delivers object, block, and file storage in one unified system
1.ceph osd pool create test 1024 1024 2.rbd -p test create myimage --size 10240 3.rbd map myimage -p test 4.mkfs.xfs -f /dev/rbd/test/myimage 5.mkdir /mnt/test 6.mount rw, /dev/rbd/test/myimage /mnt/test
I/O调度
SSD
网络
增加jumbo值 万兆网卡/交换机 网络隔离 Bond:load Infiniband
NOOP，Ohters Deadline
echo deadline > /sys/block/sdX/queue/scheduler
增大请求队列长度
echo 4096 > /sys/block/sdX/queue/nr_requests
I/O 或Buffered I/O
文件系统
XFS 4k
对齐 GPT parted mount 参数 rw,noatime,nodiratime,nobarrier,inode64,l ogbufs=8,logbsize=256k,delaylog
QEMU
Virtio
scsi 支持多队列 cache rbd cache
Controlled, Scalable, Decentralized Placement of Replicated Data
条带化
stripe_unit = 1048576 stripe_count = 4 Order = 22 size = 4M* 8 = 32M
Erasure Coding
块存储 Qemu RBD
<disk type='network' device='disk'> <driver name='qemu' type='raw' cache='none'/> <auth username='compute'> <secret type='ceph' uuid='a5d0dd94-57c4-ae55-ffe0-7e3732a24455'/> </auth> <source protocol='rbd' name='test/myimage'> <host name='192.168.0.2' port='6789'/> <host name='192.168.0.3' port='6789'/> <host name='192.168.0.6' port='6789'/> </source> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk>
写流程
Rebalance
Cache Pool Tiering
文件存储
文件存储
文件存储
对象存储
对象的写入、读取、复制、更新、删除、访问许可设置、 元数据读取或更新等 REST API, Put ,Get, Post, Delete S3 and Swift API
块存储 linux Kernel RBD
Snapshot
Ceph逐层优化
Qemu RBD Linux Kernel RBD
文件读/写命令 文件读/写命令 文件读/写命令 文件系统 文件系统 文件系统 page cache page cache I/O调度 page cache I/O调度 block device TCP block device QEMU Librbd TCP I/O调度 block device Disk Controller Disk I/O调度 block device Disk Controller Disk page cache 文件系统 文件读/写命令
OSD NODE
Ceph OSD
OSD NODE
Ceph OSD
优化参数
客户端读/写程序
选择使用系统调用或c接口
libaio+Direct 关闭页交换
page cache
vm.swappiness = 0 调整缓存大小
$ echo 5 > /proc/sys/vm/dirty_background_ratio $ echo 10 > /proc/sys/Sage Weil
started the open source Ceph project in 2004 for his doctoral dissertation at the University of California, Santa Cruz
• Inktank
Inktank was a professional services and support company for the open source Ceph file system in 2012
• RedHat
30 April 2014,Red Hat acquire Inktank
Storage for $175 Million
Ceph结构
Ceph代码调用栈
RADOS (Reliable, Autonomic Distributed Object Store)
Ceph网络
文件到osd映射