JuiceFS性能评估
原创大约 3 分钟
主要是对云原生分布式文件系统JuiceFS的评估记录，官网地址
官网介绍是一个高性能、云原生的分布式文件系统，特别是支持S3协议进行后端存储，一个机会接触到了这个项目，估想作为一个高性价比存储的一个方案。S3存储相当廉价，以腾讯云为例：
- 如果使用的是文件存储CFS，存储价格为0.35/GiB/月
- 使用对象存储COS-标准存储，存储价格为0.118/GiB/月
可以看出，如果数据能通过分层的方式下沉到对象存储，可以将成本压缩到近1/3，考虑到由于录播系统冗余节点的设计，在存储方面有很大比例的开销，这个成本的优化是值得尝试的
性能评估
性能评估其实最适合的起点应该是现对目标对接的底层存储进行测试（而不是我这样先构建文件系统后测试），当然这个只针对你需要使用对象存储作为后端存储的情况
juicefs objbench就是进行这项操作的工具，对于存储目标，我分别使用了两个环境：
- 腾讯云对象存储COS
- 本地MinIO单节点实例，存储盘为4块Btrfs RAID 0软阵列
在测试途中，发现objbench默认使用的4个thread配置，会导致COS在调用列出接口时触发限速导致无法进行，因此仅测试thread为1-3的情况，测试文件大小等参数均不会做调整，结果按照上述的两个环境顺序列出
- Thread 1
Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 1
+--------------------+-------------------+------------------+
| ITEM | VALUE | COST |
+--------------------+-------------------+------------------+
| upload objects | 31.21 MiB/s | 128.17 ms/object |
| download objects | 43.90 MiB/s | 91.12 ms/object |
| put small objects | 16.7 objects/s | 59.83 ms/object |
| get small objects | 19.2 objects/s | 52.18 ms/object |
| list objects | 1127.61 objects/s | 88.68 ms/op |
| head objects | 47.6 objects/s | 20.99 ms/object |
| delete objects | 36.4 objects/s | 27.44 ms/object |
| change permissions | not support | not support |
| change owner/group | not support | not support |
| update mtime | not support | not support |
+--------------------+-------------------+------------------+Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 1
+--------------------+-------------------+------------------+
| ITEM | VALUE | COST |
+--------------------+-------------------+------------------+
| upload objects | 21.92 MiB/s | 182.47 ms/object |
| download objects | 449.71 MiB/s | 8.89 ms/object |
| put small objects | 10.7 objects/s | 93.06 ms/object |
| get small objects | 410.5 objects/s | 2.44 ms/object |
| list objects | 5073.17 objects/s | 19.71 ms/op |
| head objects | 590.2 objects/s | 1.69 ms/object |
| delete objects | 389.0 objects/s | 2.57 ms/object |
| change permissions | not support | not support |
| change owner/group | not support | not support |
| update mtime | not support | not support |
+--------------------+-------------------+------------------+- Thread 2
Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 2
+--------------------+-------------------+------------------+
| ITEM | VALUE | COST |
+--------------------+-------------------+------------------+
| upload objects | 61.30 MiB/s | 130.50 ms/object |
| download objects | 91.44 MiB/s | 87.49 ms/object |
| put small objects | 32.3 objects/s | 61.88 ms/object |
| get small objects | 42.1 objects/s | 47.56 ms/object |
| list objects | 2577.70 objects/s | 77.59 ms/op |
| head objects | 109.8 objects/s | 18.22 ms/object |
| delete objects | 71.7 objects/s | 27.90 ms/object |
| change permissions | not support | not support |
| change owner/group | not support | not support |
| update mtime | not support | not support |
+--------------------+-------------------+------------------+Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 2
+--------------------+--------------------+------------------+
| ITEM | VALUE | COST |
+--------------------+--------------------+------------------+
| upload objects | 28.39 MiB/s | 281.82 ms/object |
| download objects | 866.93 MiB/s | 9.23 ms/object |
| put small objects | 13.6 objects/s | 147.34 ms/object |
| get small objects | 935.0 objects/s | 2.14 ms/object |
| list objects | 10045.11 objects/s | 19.91 ms/op |
| head objects | 1136.7 objects/s | 1.76 ms/object |
| delete objects | 678.2 objects/s | 2.95 ms/object |
| change permissions | not support | not support |
| change owner/group | not support | not support |
| update mtime | not support | not support |
+--------------------+--------------------+------------------+- Thread 3
Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 3
+--------------------+-------------------+------------------+
| ITEM | VALUE | COST |
+--------------------+-------------------+------------------+
| upload objects | 90.98 MiB/s | 131.90 ms/object |
| download objects | 134.54 MiB/s | 89.19 ms/object |
| put small objects | 50.5 objects/s | 59.36 ms/object |
| get small objects | 59.2 objects/s | 50.68 ms/object |
| list objects | 3469.11 objects/s | 86.48 ms/op |
| head objects | 139.3 objects/s | 21.54 ms/object |
| delete objects | 111.2 objects/s | 26.98 ms/object |
| change permissions | not support | not support |
| change owner/group | not support | not support |
| update mtime | not support | not support |
+--------------------+-------------------+------------------+Benchmark finished! block-size: 4.0 MiB, big-object-size: 1.0 GiB, small-object-size: 128 KiB, small-objects: 100, NumThreads: 3
+--------------------+--------------------+------------------+
| ITEM | VALUE | COST |
+--------------------+--------------------+------------------+
| upload objects | 34.74 MiB/s | 345.38 ms/object |
| download objects | 1210.06 MiB/s | 9.92 ms/object |
| put small objects | 19.8 objects/s | 151.14 ms/object |
| get small objects | 1388.2 objects/s | 2.16 ms/object |
| list objects | 11812.61 objects/s | 25.40 ms/op |
| head objects | 1680.3 objects/s | 1.79 ms/object |
| delete objects | 1147.5 objects/s | 2.61 ms/object |
| change permissions | not support | not support |
| change owner/group | not support | not support |
| update mtime | not support | not support |
+--------------------+--------------------+------------------+