momo's Blog.

对象存储,NAS,本地盘对比

字数统计: 8.5k阅读时长: 51 min
2023/05/25 Share

前言

对于Pod落地的BI日志, 有几种处理方法.

  1. 直接落本地磁盘
  2. 挂载NAS
  3. 挂载对象存储

首先很明显, 这3个存储方式, 性能从高到低是 1 > 2 > 3。而站在需求的角度上考虑, 首先是在弹性伸缩的前提下, 如何保证BI的日志不丢, 如果丢了需要保留原始日志做补传。那日志肯定是需要保留的, 而集群的弹性伸缩, node肯定是要删掉的。所以落本地磁盘就有几个问题。

  • 如何保证日志收集完整上传日志以后才删除node.
  • 如何保留源日志，以便出现问题能够补发之类的操作.

这次主要是针对于3种类型磁盘, 简单做一个对比.

OSS 测试

写入

  1. 使用 1 MB 的 I/O 块大小和至少 64 的 I/O 深度，通过执行具有多个（16 个或更多）并行数据流的顺序写入来测试写入吞吐量：
    1
    2
    3
    4
    5
    fio --name=write_throughput --directory=oss-test --numjobs=16 \
    --size=4G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
    --direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write \
    --group_reporting=1 --iodepth_batch_submit=64 \
    --iodepth_batch_complete_max=64

测试结果

  • 任务数量 1

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    write_throughput: Laying out IO file (1 file / 4096MiB)
    Jobs: 1 (f=1): [f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]    
    write_throughput: (groupid=0, jobs=1): err= 0: pid=2381: Thu May 25 12:51:09 2023
      write: IOPS=139, BW=140MiB/s (147MB/s)(8448MiB/60218msec)
        slat (msec): min=187, max=1194, avg=456.97, stdev=112.05
        clat (nsec): min=2622, max=12165, avg=7049.83, stdev=1409.41
         lat (msec): min=187, max=1194, avg=458.17, stdev=111.63
        clat percentiles (nsec):
         |  1.00th=[ 3280],  5.00th=[ 5408], 10.00th=[ 5920], 20.00th=[ 6112],
         | 30.00th=[ 6304], 40.00th=[ 6432], 50.00th=[ 6688], 60.00th=[ 7072],
         | 70.00th=[ 7456], 80.00th=[ 7904], 90.00th=[ 8768], 95.00th=[ 9536],
         | 99.00th=[12096], 99.50th=[12224], 99.90th=[12224], 99.95th=[12224],
         | 99.99th=[12224]
       bw (  KiB/s): min=130810, max=262144, per=100.00%, avg=148014.19, stdev=44173.75, samples=116
       iops        : min=  127, max=  256, avg=144.53, stdev=43.15, samples=116
      lat (usec)   : 4=1.49%, 10=94.00%, 20=4.51%
      cpu          : usr=1.40%, sys=0.19%, ctx=67094, majf=0, minf=4
      IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=116.0%
         submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
         complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
         issued rwts: total=0,8384,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=64

    Run status group 0 (all jobs):
      WRITE: bw=140MiB/s (147MB/s), 140MiB/s-140MiB/s (147MB/s-147MB/s), io=8448MiB (8858MB), run=60218-60218msec

  • 任务数量 16

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    Jobs: 7 (f=7): [f(3),_(4),f(1),_(1),f(2),_(4),f(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]                         
    write_throughput: (groupid=0, jobs=16): err= 0: pid=646: Thu May 25 08:05:26 2023
      write: IOPS=118, BW=133MiB/s (140MB/s)(9216MiB/69119msec)
        slat (msec): min=1899, max=14210, avg=7683.33, stdev=1646.84
        clat (nsec): min=2149, max=21503, avg=9514.00, stdev=2354.12
         lat (msec): min=6406, max=14210, avg=8054.36, stdev=1140.92
        clat percentiles (nsec):
         |  1.00th=[ 3408],  5.00th=[ 6624], 10.00th=[ 6944], 20.00th=[ 7776],
         | 30.00th=[ 8384], 40.00th=[ 8768], 50.00th=[ 9408], 60.00th=[ 9792],
         | 70.00th=[10048], 80.00th=[10816], 90.00th=[12096], 95.00th=[13376],
         | 99.00th=[16064], 99.50th=[21376], 99.90th=[21376], 99.95th=[21376],
         | 99.99th=[21376]
       bw (  KiB/s): min=88086, max=131072, per=94.70%, avg=129298.54, stdev=7142.76, samples=128
       iops        : min=   86, max=  128, avg=126.17, stdev= 7.08, samples=128
      lat (usec)   : 4=1.38%, 10=65.55%, 20=32.30%, 50=0.77%
      cpu          : usr=0.09%, sys=0.03%, ctx=65557, majf=0, minf=37
      IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=127.3%
         submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
         complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
         issued rwts: total=0,8192,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=64

    Run status group 0 (all jobs):
      WRITE: bw=133MiB/s (140MB/s), 133MiB/s-133MiB/s (140MB/s-140MB/s), io=9216MiB (9664MB), run=69119-69119msec
  1. 使用 4 KB 的 I/O 块大小和至少 256 的 I/O 深度，通过执行随机写入来测试写入 IOPS
1
2
3
4
fio --name=write_iops --directory=./fio-random --size=4G \
--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
--verify=0 --bs=4K --iodepth=256 --rw=randwrite --group_reporting=1  \
--iodepth_batch_submit=256  --iodepth_batch_complete_max=256

  • 任务数量 1

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    write_iops: (groupid=0, jobs=1): err= 0: pid=729: Thu May 25 08:12:07 2023
      write: IOPS=426, BW=1722KiB/s (1763kB/s)(102MiB/60671msec)
        slat (msec): min=153, max=1230, avg=596.26, stdev=325.83
        clat (nsec): min=2231, max=44668, avg=8521.19, stdev=3706.08
         lat (msec): min=167, max=1230, avg=600.65, stdev=324.44
        clat percentiles (nsec):
         |  1.00th=[ 6816],  5.00th=[ 7072], 10.00th=[ 7328], 20.00th=[ 7520],
         | 30.00th=[ 7776], 40.00th=[ 7904], 50.00th=[ 8096], 60.00th=[ 8256],
         | 70.00th=[ 8512], 80.00th=[ 8768], 90.00th=[ 9280], 95.00th=[ 9536],
         | 99.00th=[11072], 99.50th=[44800], 99.90th=[44800], 99.95th=[44800],
         | 99.99th=[44800]
       bw (  KiB/s): min= 2043, max= 6144, per=100.00%, avg=2522.23, stdev=1032.89, samples=82
       iops        : min=  510, max= 1536, avg=630.51, stdev=258.24, samples=82
      lat (usec)   : 4=0.39%, 10=96.65%, 20=1.97%, 50=0.99%
      cpu          : usr=0.09%, sys=0.30%, ctx=25856, majf=0, minf=4
      IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=124.8%
         submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=100.0%
         complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=100.0%
         issued rwts: total=0,25856,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=256

    Run status group 0 (all jobs):
      WRITE: bw=1722KiB/s (1763kB/s), 1722KiB/s-1722KiB/s (1763kB/s-1763kB/s), io=102MiB (107MB), run=60671-60671msec

  • 任务数量16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
write_iops: (groupid=0, jobs=16): err= 1 (file:filesetup.c:137, func=unlink, error=Operation not permitted): pid=0: Thu May 25 08:35:50 2023
  write: IOPS=2111, BW=8695KiB/s (8904kB/s)(527MiB/62062msec)
    slat (msec): min=364, max=3514, avg=1752.15, stdev=875.87
    clat (nsec): min=1798, max=41056, avg=17675.70, stdev=4035.85
     lat (msec): min=437, max=3514, avg=1790.34, stdev=859.28
    clat percentiles (nsec):
     |  1.00th=[11328],  5.00th=[12224], 10.00th=[12992], 20.00th=[14144],
     | 30.00th=[15040], 40.00th=[15936], 50.00th=[17792], 60.00th=[18816],
     | 70.00th=[20096], 80.00th=[20864], 90.00th=[21888], 95.00th=[22912],
     | 99.00th=[30080], 99.50th=[31616], 99.90th=[41216], 99.95th=[41216],
     | 99.99th=[41216]
   bw (  KiB/s): min= 2035, max= 4096, per=24.13%, avg=2097.94, stdev=312.94, samples=499
   iops        : min=  508, max= 1024, avg=524.40, stdev=78.22, samples=499
  lat (usec)   : 2=0.01%, 4=0.28%, 10=0.10%, 20=69.65%, 50=29.96%
  cpu          : usr=0.01%, sys=0.14%, ctx=131115, majf=0, minf=36
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=129.1%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=100.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=100.0%
     issued rwts: total=0,131072,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=8695KiB/s (8904kB/s), 8695KiB/s-8695KiB/s (8904kB/s-8904kB/s), io=527MiB (553MB), run=62062-62062msec

读取

  1. 使用 1 MB 的 I/O 块大小和至少 64 的 I/O 深度，通过执行具有多个（16 个或更多）并行数据流的顺序读取来测试读取吞吐量：
1
2
3
4
5
fio --name=read_throughput --directory=./oss-read --numjobs=16 \
--size=4G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read \
--group_reporting=1 \
--iodepth_batch_submit=64 --iodepth_batch_complete_max=64
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
read_throughput: (groupid=0, jobs=16): err= 0: pid=1481: Thu May 25 09:11:12 2023
   read: IOPS=96, BW=107MiB/s (112MB/s)(10.1GiB/97125msec)
    slat (msec): min=27, max=74597, avg=15126.64, stdev=20604.50
    clat (usec): min=2, max=203, avg=10.75, stdev=16.51
     lat (msec): min=27, max=43774, avg=8812.90, stdev=8207.42
    clat percentiles (usec):
     |  1.00th=[    4],  5.00th=[    6], 10.00th=[    7], 20.00th=[    8],
     | 30.00th=[    8], 40.00th=[    9], 50.00th=[    9], 60.00th=[   10],
     | 70.00th=[   11], 80.00th=[   11], 90.00th=[   13], 95.00th=[   14],
     | 99.00th=[   47], 99.50th=[  204], 99.90th=[  204], 99.95th=[  204],
     | 99.99th=[  204]
   bw (  KiB/s): min=89775, max=786432, per=100.00%, avg=135918.41, stdev=60238.62, samples=139
   iops        : min=   87, max=  768, avg=132.64, stdev=58.85, samples=139
  lat (usec)   : 4=1.23%, 10=65.70%, 20=30.35%, 50=2.03%, 250=0.68%
  cpu          : usr=0.00%, sys=0.07%, ctx=74801, majf=0, minf=53
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=111.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
     issued rwts: total=9344,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s), io=10.1GiB (10.9GB), run=97125-97125msec

1 job

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
read_throughput: (groupid=0, jobs=1): err= 0: pid=2524: Thu May 25 13:02:20 2023
   read: IOPS=1229, BW=1231MiB/s (1291MB/s)(72.1GiB/60007msec)
    slat (msec): min=24, max=6491, avg=53.22, stdev=315.61
    clat (nsec): min=1002, max=22007, avg=4562.26, stdev=1523.10
     lat (msec): min=24, max=6491, avg=52.03, stdev=313.16
    clat percentiles (nsec):
     |  1.00th=[ 2416],  5.00th=[ 3408], 10.00th=[ 3632], 20.00th=[ 3920],
     | 30.00th=[ 4128], 40.00th=[ 4256], 50.00th=[ 4384], 60.00th=[ 4512],
     | 70.00th=[ 4704], 80.00th=[ 4896], 90.00th=[ 5216], 95.00th=[ 5664],
     | 99.00th=[ 9792], 99.50th=[17280], 99.90th=[20352], 99.95th=[21888],
     | 99.99th=[21888]
   bw (  MiB/s): min=  128, max= 2560, per=100.00%, avg=1994.19, stdev=773.74, samples=74
   iops        : min=  128, max= 2560, avg=1994.18, stdev=773.73, samples=74
  lat (usec)   : 2=0.32%, 4=22.23%, 10=76.51%, 20=0.78%, 50=0.17%
  cpu          : usr=0.49%, sys=7.71%, ctx=590341, majf=0, minf=3
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.9%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=100.0%, >=64=0.0%
     issued rwts: total=73792,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1231MiB/s (1291MB/s), 1231MiB/s-1231MiB/s (1291MB/s-1291MB/s), io=72.1GiB (77.4GB), run=60007-60007msec
  1. 使用 4 KB 的 I/O 块大小和至少 256 的 I/O 深度，通过执行随机读取来测试读取 IOPS:
1
2
3
4
fio --name=read_iops --directory=./oss-read --size=4G \
--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
--verify=0 --bs=4K --iodepth=256 --rw=randread --group_reporting=1 \
--iodepth_batch_submit=256  --iodepth_batch_complete_max=256
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
read_iops: (groupid=0, jobs=1): err= 0: pid=345: Thu May 25 10:02:24 2023
   read: IOPS=42.4k, BW=166MiB/s (174MB/s)(9946MiB/60005msec)
    slat (msec): min=3, max=36562, avg= 9.66, stdev=367.02
    clat (nsec): min=546, max=25947, avg=6419.54, stdev=2750.97
     lat (msec): min=3, max=1622, avg= 6.00, stdev=18.78
    clat percentiles (nsec):
     |  1.00th=[ 3504],  5.00th=[ 3632], 10.00th=[ 3760], 20.00th=[ 3984],
     | 30.00th=[ 4256], 40.00th=[ 4704], 50.00th=[ 5408], 60.00th=[ 6496],
     | 70.00th=[ 7968], 80.00th=[ 9024], 90.00th=[10304], 95.00th=[11072],
     | 99.00th=[14016], 99.50th=[16320], 99.90th=[20608], 99.95th=[22656],
     | 99.99th=[25984]
   bw (  KiB/s): min= 2048, max=221184, per=100.00%, avg=174064.21, stdev=55855.91, samples=117
   iops        : min=  512, max=55296, avg=43516.03, stdev=13963.98, samples=117
  lat (nsec)   : 750=0.02%, 1000=0.07%
  lat (usec)   : 2=0.17%, 4=20.24%, 10=67.50%, 20=11.84%, 50=0.17%
  cpu          : usr=11.06%, sys=17.94%, ctx=2545949, majf=0, minf=3
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=100.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=100.0%
     issued rwts: total=2545920,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=166MiB/s (174MB/s), 166MiB/s-166MiB/s (174MB/s-174MB/s), io=9946MiB (10.4GB), run=60005-60005msec
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
read_iops: (groupid=0, jobs=5): err= 0: pid=265: Thu May 25 09:57:59 2023
   read: IOPS=4925, BW=19.3MiB/s (20.3MB/s)(1165MiB/60293msec)
    slat (msec): min=40, max=148378, avg=881.26, stdev=9478.51
    clat (nsec): min=1558, max=584833, avg=10824.22, stdev=17063.64
     lat (msec): min=40, max=3556, avg=259.15, stdev=194.75
    clat percentiles (usec):
     |  1.00th=[    7],  5.00th=[    8], 10.00th=[    8], 20.00th=[    9],
     | 30.00th=[    9], 40.00th=[   10], 50.00th=[   10], 60.00th=[   11],
     | 70.00th=[   12], 80.00th=[   13], 90.00th=[   14], 95.00th=[   16],
     | 99.00th=[   21], 99.50th=[   25], 99.90th=[   37], 99.95th=[  586],
     | 99.99th=[  586]
   bw (  KiB/s): min= 1426, max=22528, per=20.80%, avg=4115.34, stdev=2466.49, samples=570
   iops        : min=  356, max= 5632, avg=1028.79, stdev=616.61, samples=570
  lat (usec)   : 2=0.01%, 4=0.37%, 10=58.66%, 20=39.93%, 50=0.94%
  lat (usec)   : 750=0.09%
  cpu          : usr=0.14%, sys=0.68%, ctx=297073, majf=0, minf=13
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.4%
     submit    : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=100.0%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=100.0%
     issued rwts: total=296960,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=19.3MiB/s (20.3MB/s), 19.3MiB/s-19.3MiB/s (20.3MB/s-20.3MB/s), io=1165MiB (1222MB), run=60293-60293msec

ESSD磁盘

阿里云计算型主机, 已经不能挂载普通云盘，所以只能拿SSD做测试.

100GB 总量

顺序写入

  • 任务数 1

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    write_throughput: (groupid=0, jobs=1): err= 0: pid=10150: Thu May 25 17:00:44 2023
      write: IOPS=124, BW=126MiB/s (132MB/s)(7564MiB/60250msec); 0 zone resets
        slat (usec): min=30, max=392485, avg=7982.01, stdev=11484.74
        clat (msec): min=16, max=3178, avg=504.04, stdev=159.30
         lat (msec): min=34, max=3197, avg=512.02, stdev=159.51
        clat percentiles (msec):
         |  1.00th=[  257],  5.00th=[  351], 10.00th=[  443], 20.00th=[  493],
         | 30.00th=[  502], 40.00th=[  502], 50.00th=[  502], 60.00th=[  510],
         | 70.00th=[  510], 80.00th=[  510], 90.00th=[  518], 95.00th=[  523],
         | 99.00th=[  776], 99.50th=[ 1737], 99.90th=[ 2601], 99.95th=[ 3004],
         | 99.99th=[ 3171]
       bw (  KiB/s): min=98304, max=190464, per=99.58%, avg=128017.07, stdev=10640.05, samples=120
       iops        : min=   96, max=  186, avg=125.02, stdev=10.39, samples=120
      lat (msec)   : 20=0.01%, 50=0.08%, 100=0.25%, 250=0.59%, 500=38.16%
      lat (msec)   : 750=60.69%, 1000=0.15%, 2000=0.56%, >=2000=0.35%
      cpu          : usr=0.77%, sys=0.11%, ctx=3335, majf=0, minf=58
      IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=99.8%
         submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
         complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         issued rwts: total=0,7500,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=64

    Run status group 0 (all jobs):
      WRITE: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=7564MiB (7931MB), run=60250-60250msec

    Disk stats (read/write):
      nvme0n1: ios=0/31669, merge=0/50, ticks=0/8117333, in_queue=8117333, util=99.88%

  • 任务数 16

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    write_throughput: (groupid=0, jobs=16): err= 0: pid=8207: Thu May 25 16:56:43 2023
      write: IOPS=147, BW=164MiB/s (172MB/s)(10.3GiB/64086msec); 0 zone resets
        slat (usec): min=38, max=22862k, avg=928750.84, stdev=3152457.07
        clat (usec): min=179, max=24708k, avg=5941143.86, stdev=4914369.56
         lat (msec): min=17, max=32514, avg=6858.49, stdev=5822.58
        clat percentiles (msec):
         |  1.00th=[  209],  5.00th=[  502], 10.00th=[  558], 20.00th=[ 1200],
         | 30.00th=[ 1989], 40.00th=[ 3339], 50.00th=[ 5000], 60.00th=[ 6477],
         | 70.00th=[ 8087], 80.00th=[10000], 90.00th=[12953], 95.00th=[15637],
         | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
         | 99.99th=[17113]
       bw (  KiB/s): min=24344, max=1349169, per=100.00%, avg=245712.02, stdev=19220.22, samples=1185
       iops        : min=   19, max= 1317, avg=239.52, stdev=18.76, samples=1185
      lat (usec)   : 250=0.03%, 500=0.07%, 750=0.07%, 1000=0.06%
      lat (msec)   : 2=0.08%, 20=0.01%, 50=0.08%, 100=0.18%, 250=0.65%
      lat (msec)   : 500=4.20%, 750=8.02%, 1000=4.24%, 2000=15.60%, >=2000=77.33%
      cpu          : usr=0.07%, sys=0.01%, ctx=6070, majf=0, minf=945
      IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=95.8%
         submit    : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
         complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.2%, >=64=0.0%
         issued rwts: total=0,9482,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=64

    Run status group 0 (all jobs):
      WRITE: bw=164MiB/s (172MB/s), 164MiB/s-164MiB/s (172MB/s-172MB/s), io=10.3GiB (11.0GB), run=64086-64086msec

    Disk stats (read/write):
      nvme0n1: ios=0/42211, merge=0/1316, ticks=0/19013423, in_queue=19013423, util=99.91%

随机写

  • 任务 1 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    write_iops: (groupid=0, jobs=1): err= 0: pid=11172: Thu May 25 17:03:06 2023
      write: IOPS=2986, BW=11.7MiB/s (12.2MB/s)(702MiB/60080msec); 0 zone resets
        slat (usec): min=3, max=149890, avg=34096.87, stdev=13557.29
        clat (usec): min=2, max=1417.6k, avg=48246.70, stdev=49087.82
         lat (msec): min=8, max=1437, avg=82.34, stdev=49.48
        clat percentiles (usec):
         |  1.00th=[      4],  5.00th=[  19792], 10.00th=[  30278],
         | 20.00th=[  40109], 30.00th=[  40109], 40.00th=[  40109],
         | 50.00th=[  40633], 60.00th=[  49546], 70.00th=[  50070],
         | 80.00th=[  50070], 90.00th=[  60031], 95.00th=[  69731],
         | 99.00th=[ 110625], 99.50th=[ 320865], 99.90th=[ 918553],
         | 99.95th=[1010828], 99.99th=[1283458]
       bw (  KiB/s): min= 9512, max=12912, per=100.00%, avg=11966.60, stdev=545.75, samples=120
       iops        : min= 2378, max= 3228, avg=2991.65, stdev=136.44, samples=120
      lat (usec)   : 4=1.31%, 10=1.97%, 20=0.07%, 50=0.06%, 100=0.05%
      lat (usec)   : 250=0.14%, 500=0.03%
      lat (msec)   : 10=0.44%, 20=1.20%, 50=75.06%, 100=18.68%, 250=0.45%
      lat (msec)   : 500=0.29%, 750=0.16%, 1000=0.07%, 2000=0.07%
      cpu          : usr=0.14%, sys=1.57%, ctx=14436, majf=0, minf=58
      IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
         submit    : 0=0.0%, 4=2.2%, 8=2.1%, 16=5.2%, 32=9.8%, 64=23.9%, >=64=56.8%
         complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.5%, 32=2.6%, 64=15.0%, >=64=81.9%
         issued rwts: total=0,179423,0,0 short=0,0,0,0 dropped=0,0,0,0
         latency   : target=0, window=0, percentile=100.00%, depth=256

    Run status group 0 (all jobs):
      WRITE: bw=11.7MiB/s (12.2MB/s), 11.7MiB/s-11.7MiB/s (12.2MB/s-12.2MB/s), io=702MiB (736MB), run=60080-60080msec

    Disk stats (read/write):
      nvme0n1: ios=0/188894, merge=0/11801, ticks=0/8296631, in_queue=8296632, util=99.92%

  • 任务 16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
write_iops: (groupid=0, jobs=16): err= 0: pid=12485: Thu May 25 17:05:42 2023
  write: IOPS=3077, BW=12.3MiB/s (12.9MB/s)(785MiB/64061msec); 0 zone resets
    slat (usec): min=4, max=16561k, avg=746267.07, stdev=1672982.14
    clat (usec): min=2, max=22774k, avg=425818.67, stdev=1220008.09
     lat (msec): min=35, max=28480, avg=1138.31, stdev=2098.96
    clat percentiles (usec):
     |  1.00th=[       5],  5.00th=[       7], 10.00th=[      10],
     | 20.00th=[      16], 30.00th=[   29754], 40.00th=[   40109],
     | 50.00th=[   49546], 60.00th=[   50070], 70.00th=[   69731],
     | 80.00th=[  400557], 90.00th=[ 1317012], 95.00th=[ 2298479],
     | 99.00th=[ 5133829], 99.50th=[ 8925479], 99.90th=[14294189],
     | 99.95th=[16307454], 99.99th=[17112761]
   bw (  KiB/s): min=17398, max=152384, per=100.00%, avg=47604.91, stdev=2234.19, samples=520
   iops        : min= 4349, max=38096, avg=11900.98, stdev=558.55, samples=520
  lat (usec)   : 4=0.64%, 10=10.69%, 20=12.08%, 50=1.25%, 100=0.20%
  lat (usec)   : 250=0.53%, 500=0.14%
  lat (msec)   : 10=0.99%, 20=1.90%, 50=29.53%, 100=16.85%, 250=3.00%
  lat (msec)   : 500=4.27%, 750=3.29%, 1000=2.72%, 2000=6.29%, >=2000=6.06%
  cpu          : usr=0.01%, sys=0.12%, ctx=22231, majf=0, minf=931
  IO depths    : 1=0.0%, 2=0.1%, 4=0.0%, 8=0.4%, 16=1.8%, 32=3.9%, >=64=93.7%
     submit    : 0=0.0%, 4=2.0%, 8=1.7%, 16=4.0%, 32=9.0%, 64=14.0%, >=64=69.3%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.2%, 32=0.2%, 64=0.6%, >=64=99.1%
     issued rwts: total=0,197179,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=12.3MiB/s (12.9MB/s), 12.3MiB/s-12.3MiB/s (12.9MB/s-12.9MB/s), io=785MiB (823MB), run=64061-64061msec

Disk stats (read/write):
  nvme0n1: ios=82/208138, merge=0/16659, ticks=24399/16575521, in_queue=16599920, util=99.92%

顺序读

job 1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
read_throughput: (groupid=0, jobs=1): err= 0: pid=126765: Thu May 25 21:33:59 2023
  read: IOPS=124, BW=126MiB/s (132MB/s)(7589MiB/60455msec)
    slat (usec): min=8, max=281175, avg=205029.47, stdev=69730.89
    clat (usec): min=2, max=3108.5k, avg=283186.16, stdev=141346.21
     lat (msec): min=39, max=3358, avg=488.09, stdev=143.09
    clat percentiles (usec):
     |  1.00th=[      4],  5.00th=[ 200279], 10.00th=[ 250610],
     | 20.00th=[ 250610], 30.00th=[ 258999], 40.00th=[ 258999],
     | 50.00th=[ 261096], 60.00th=[ 270533], 70.00th=[ 270533],
     | 80.00th=[ 287310], 90.00th=[ 400557], 95.00th=[ 421528],
     | 99.00th=[ 488637], 99.50th=[1019216], 99.90th=[2399142],
     | 99.95th=[2868904], 99.99th=[3103785]
   bw (  KiB/s): min=65536, max=174080, per=99.91%, avg=128424.48, stdev=19948.61, samples=120
   iops        : min=   64, max=  170, avg=125.41, stdev=19.48, samples=120
  lat (usec)   : 4=1.34%, 10=0.97%
  lat (msec)   : 20=0.12%, 50=0.32%, 100=0.73%, 250=8.29%, 500=87.79%
  lat (msec)   : 750=0.28%, 1000=0.07%, 2000=0.37%, >=2000=0.15%
  cpu          : usr=0.00%, sys=0.11%, ctx=3754, majf=0, minf=58
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=97.0%, >=64=2.6%
     submit    : 0=0.0%, 4=13.6%, 8=9.7%, 16=19.0%, 32=43.6%, 64=14.1%, >=64=0.0%
     complete  : 0=0.0%, 4=9.0%, 8=4.0%, 16=5.1%, 32=43.0%, 64=39.0%, >=64=0.0%
     issued rwts: total=7525,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=126MiB/s (132MB/s), 126MiB/s-126MiB/s (132MB/s-132MB/s), io=7589MiB (7958MB), run=60455-60455msec

Disk stats (read/write):
  nvme0n1: ios=31393/13, merge=0/11, ticks=7990985/2254, in_queue=7993240, util=99.92%

job 16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
read_throughput: (groupid=0, jobs=16): err= 0: pid=123513: Thu May 25 21:26:37 2023
  read: IOPS=120, BW=135MiB/s (142MB/s)(9505MiB/70417msec)
    slat (usec): min=9, max=17282k, avg=5162367.20, stdev=4045516.86
    clat (usec): min=3, max=23984k, avg=1992130.61, stdev=3492503.64
     lat (msec): min=279, max=28852, avg=6696.72, stdev=5060.50
    clat percentiles (usec):
     |  1.00th=[       5],  5.00th=[       6], 10.00th=[       7],
     | 20.00th=[       8], 30.00th=[       9], 40.00th=[      10],
     | 50.00th=[  240124], 60.00th=[  530580], 70.00th=[ 1719665],
     | 80.00th=[ 3674211], 90.00th=[ 7012877], 95.00th=[10133439],
     | 99.00th=[15636366], 99.50th=[17112761], 99.90th=[17112761],
     | 99.95th=[17112761], 99.99th=[17112761]
   bw (  MiB/s): min= 1309, max= 1932, per=100.00%, avg=1742.76, stdev=12.20, samples=161
   iops        : min= 1302, max= 1932, avg=1741.62, stdev=12.33, samples=161
  lat (usec)   : 4=0.19%, 10=41.13%, 20=5.57%
  lat (msec)   : 20=0.54%, 50=0.54%, 100=0.55%, 250=2.94%, 500=7.94%
  lat (msec)   : 750=3.15%, 1000=3.45%, 2000=7.22%, >=2000=28.19%
  cpu          : usr=0.00%, sys=0.01%, ctx=6398, majf=0, minf=933
  IO depths    : 1=0.0%, 2=0.0%, 4=1.5%, 8=11.3%, 16=30.2%, 32=52.8%, >=64=0.0%
     submit    : 0=0.0%, 4=5.9%, 8=4.2%, 16=11.5%, 32=27.4%, 64=51.0%, >=64=0.0%
     complete  : 0=0.0%, 4=0.6%, 8=1.1%, 16=1.7%, 32=1.7%, 64=94.9%, >=64=0.0%
     issued rwts: total=8481,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=135MiB/s (142MB/s), 135MiB/s-135MiB/s (142MB/s-142MB/s), io=9505MiB (9967MB), run=70417-70417msec

Disk stats (read/write):
  nvme0n1: ios=37999/45, merge=0/29, ticks=18585189/74122, in_queue=18659312, util=99.92%

随机读

job 1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
read_iops: (groupid=0, jobs=1): err= 0: pid=25020: Thu May 25 17:33:48 2023
  read: IOPS=2997, BW=11.7MiB/s (12.3MB/s)(704MiB/60050msec)
    slat (nsec): min=1560, max=52514k, avg=34357149.12, stdev=12188575.32
    clat (nsec): min=1721, max=1489.0M, avg=47445460.62, stdev=48879182.38
     lat (msec): min=17, max=1509, avg=81.80, stdev=48.84
    clat percentiles (usec):
     |  1.00th=[      3],  5.00th=[  20055], 10.00th=[  39584],
     | 20.00th=[  40109], 30.00th=[  40109], 40.00th=[  40633],
     | 50.00th=[  47973], 60.00th=[  49546], 70.00th=[  49546],
     | 80.00th=[  50070], 90.00th=[  60031], 95.00th=[  69731],
     | 99.00th=[  80217], 99.50th=[  90702], 99.90th=[1010828],
     | 99.95th=[1166017], 99.99th=[1367344]
   bw (  KiB/s): min=11160, max=13248, per=100.00%, avg=12014.92, stdev=472.89, samples=119
   iops        : min= 2790, max= 3312, avg=3003.73, stdev=118.22, samples=119
  lat (usec)   : 2=0.39%, 4=2.60%, 10=0.28%, 20=0.02%, 250=0.06%
  lat (usec)   : 500=0.09%, 750=0.07%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.47%, 20=1.00%, 50=76.10%
  lat (msec)   : 100=18.63%, 250=0.04%, 500=0.07%, 750=0.07%, 1000=0.07%
  lat (msec)   : 2000=0.10%
  cpu          : usr=0.11%, sys=0.35%, ctx=20966, majf=0, minf=58
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.1%
     submit    : 0=0.0%, 4=1.8%, 8=2.0%, 16=4.6%, 32=9.0%, 64=23.0%, >=64=59.6%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=1.6%, 64=11.7%, >=64=86.7%
     issued rwts: total=180001,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=11.7MiB/s (12.3MB/s), 11.7MiB/s-11.7MiB/s (12.3MB/s-12.3MB/s), io=704MiB (738MB), run=60050-60050msec

Disk stats (read/write):
  nvme0n1: ios=188745/27, merge=1/10, ticks=8113044/858, in_queue=8113901, util=99.89%

job 16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
read_iops: (groupid=0, jobs=16): err= 0: pid=30079: Thu May 25 17:45:55 2023
  read: IOPS=3379, BW=13.4MiB/s (14.1MB/s)(840MiB/62464msec)
    slat (nsec): min=1808, max=12031M, avg=780953977.50, stdev=1471112704.82
    clat (nsec): min=1745, max=11459M, avg=311780802.03, stdev=906870329.87
     lat (msec): min=7, max=16250, avg=1050.40, stdev=1698.93
    clat percentiles (usec):
     |  1.00th=[       4],  5.00th=[       6], 10.00th=[       7],
     | 20.00th=[       9], 30.00th=[      13], 40.00th=[     578],
     | 50.00th=[   40109], 60.00th=[   60031], 70.00th=[  129500],
     | 80.00th=[  250610], 90.00th=[  717226], 95.00th=[ 1769997],
     | 99.00th=[ 4664067], 99.50th=[ 6140462], 99.90th=[ 9059697],
     | 99.95th=[ 9462350], 99.99th=[11475616]
   bw (  KiB/s): min=21847, max=88276, per=100.00%, avg=41241.94, stdev=1379.08, samples=633
   iops        : min= 5458, max=22068, avg=10310.23, stdev=344.76, samples=633
  lat (usec)   : 2=0.03%, 4=2.16%, 10=21.61%, 20=15.28%, 50=1.00%
  lat (usec)   : 100=0.02%, 250=0.01%, 500=0.10%, 750=0.19%, 1000=0.17%
  lat (msec)   : 2=0.01%, 10=0.48%, 20=1.91%, 50=14.57%, 100=9.79%
  lat (msec)   : 250=12.97%, 500=8.83%, 750=1.62%, 1000=1.47%, 2000=3.94%
  lat (msec)   : >=2000=4.42%
  cpu          : usr=0.01%, sys=0.03%, ctx=31086, majf=0, minf=931
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.1%, 16=1.0%, 32=10.3%, >=64=88.5%
     submit    : 0=0.0%, 4=1.9%, 8=1.4%, 16=3.6%, 32=8.2%, 64=14.8%, >=64=70.1%
     complete  : 0=0.0%, 4=0.0%, 8=0.0%, 16=0.4%, 32=2.3%, 64=3.1%, >=64=94.3%
     issued rwts: total=211113,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=13.4MiB/s (14.1MB/s), 13.4MiB/s-13.4MiB/s (14.1MB/s-14.1MB/s), io=840MiB (880MB), run=62464-62464msec

Disk stats (read/write):
  nvme0n1: ios=222273/45, merge=6/25, ticks=17603262/19768, in_queue=17623030, util=99.94%

顺序读

1
2


NAS

顺序写

job 1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
write_throughput: (groupid=0, jobs=1): err= 0: pid=1621: Thu May 25 14:21:27 2023
  write: IOPS=179, BW=181MiB/s (190MB/s)(10.7GiB/60365msec)
    slat (usec): min=55, max=2273, avg=112.54, stdev=102.02
    clat (msec): min=129, max=708, avg=354.87, stdev=38.39
     lat (msec): min=129, max=708, avg=354.99, stdev=38.37
    clat percentiles (msec):
     |  1.00th=[  213],  5.00th=[  317], 10.00th=[  338], 20.00th=[  347],
     | 30.00th=[  351], 40.00th=[  351], 50.00th=[  355], 60.00th=[  355],
     | 70.00th=[  359], 80.00th=[  359], 90.00th=[  368], 95.00th=[  397],
     | 99.00th=[  523], 99.50th=[  550], 99.90th=[  659], 99.95th=[  693],
     | 99.99th=[  701]
   bw (  KiB/s): min=126976, max=260096, per=100.00%, avg=185160.96, stdev=14745.40, samples=120
   iops        : min=  124, max=  254, avg=180.80, stdev=14.40, samples=120
  lat (msec)   : 250=1.62%, 500=97.58%, 750=1.38%
  cpu          : usr=1.04%, sys=0.84%, ctx=10210, majf=0, minf=4
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=2.4%, >=64=102.6%
     submit    : 0=0.0%, 4=99.6%, 8=0.3%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.0%
     complete  : 0=0.0%, 4=99.6%, 8=0.3%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,10851,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=181MiB/s (190MB/s), 181MiB/s-181MiB/s (190MB/s-190MB/s), io=10.7GiB (11.4GB), run=60365-60365msec

顺序写 16job

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
write_throughput: (groupid=0, jobs=16): err= 0: pid=258: Thu May 25 12:38:56 2023
  write: IOPS=136, BW=152MiB/s (159MB/s)(9.90GiB/66746msec)
    slat (usec): min=80, max=7824, avg=339.19, stdev=1091.50
    clat (msec): min=482, max=13331, avg=6767.24, stdev=1547.18
     lat (msec): min=489, max=13331, avg=6767.72, stdev=1546.43
    clat percentiles (msec):
     |  1.00th=[ 1871],  5.00th=[ 3742], 10.00th=[ 5805], 20.00th=[ 6745],
     | 30.00th=[ 6745], 40.00th=[ 6812], 50.00th=[ 6812], 60.00th=[ 6812],
     | 70.00th=[ 6812], 80.00th=[ 6812], 90.00th=[ 7013], 95.00th=[ 9731],
     | 99.00th=[12416], 99.50th=[12953], 99.90th=[13221], 99.95th=[13355],
     | 99.99th=[13355]
   bw (  KiB/s): min= 1771, max=18652, per=6.20%, avg=9639.62, stdev=1656.56, samples=1936
   iops        : min=    1, max=   18, avg= 9.36, stdev= 1.64, samples=1936
  lat (msec)   : 500=0.01%, 750=0.09%, 1000=0.16%
  cpu          : usr=0.07%, sys=0.05%, ctx=10288, majf=0, minf=46
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=108.8%
     submit    : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.2%, >=64=0.0%
     complete  : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.2%, >=64=0.0%
     issued rwts: total=0,9121,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=152MiB/s (159MB/s), 152MiB/s-152MiB/s (159MB/s-159MB/s), io=9.90GiB (10.6GB), run=66746-66746msec

随机写 job1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
write_iops: (groupid=0, jobs=1): err= 0: pid=318: Thu May 25 12:41:52 2023
  write: IOPS=14.0k, BW=58.6MiB/s (61.5MB/s)(3535MiB/60315msec)
    slat (nsec): min=949, max=1081.9k, avg=10777.69, stdev=16107.81
    clat (msec): min=4, max=448, avg=17.05, stdev=14.65
     lat (msec): min=4, max=448, avg=17.06, stdev=14.65
    clat percentiles (msec):
     |  1.00th=[    8],  5.00th=[    9], 10.00th=[   10], 20.00th=[   11],
     | 30.00th=[   12], 40.00th=[   14], 50.00th=[   15], 60.00th=[   17],
     | 70.00th=[   19], 80.00th=[   21], 90.00th=[   24], 95.00th=[   28],
     | 99.00th=[   68], 99.50th=[  101], 99.90th=[  222], 99.95th=[  234],
     | 99.99th=[  334]
   bw (  KiB/s): min=31392, max=84224, per=100.00%, avg=60306.31, stdev=8377.83, samples=120
   iops        : min= 7848, max=21056, avg=15076.55, stdev=2094.46, samples=120
  lat (msec)   : 10=13.45%, 20=65.38%, 50=19.15%, 100=1.54%, 250=0.47%
  lat (msec)   : 500=0.03%
  cpu          : usr=1.57%, sys=6.44%, ctx=398014, majf=0, minf=3
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=104.1%
     submit    : 0=0.0%, 4=94.0%, 8=4.4%, 16=1.4%, 32=0.2%, 64=0.1%, >=64=0.1%
     complete  : 0=0.0%, 4=93.9%, 8=4.4%, 16=1.4%, 32=0.2%, 64=0.1%, >=64=0.1%
     issued rwts: total=0,904682,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=58.6MiB/s (61.5MB/s), 58.6MiB/s-58.6MiB/s (61.5MB/s-61.5MB/s), io=3535MiB (3707MB), run=60315-60315msec

随机写 job 16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
write_iops: (groupid=0, jobs=16): err= 0: pid=375: Thu May 25 12:44:15 2023
  write: IOPS=15.1k, BW=59.2MiB/s (62.1MB/s)(3571MiB/60282msec)
    slat (nsec): min=1015, max=845174, avg=4941.07, stdev=6834.25
    clat (msec): min=183, max=1718, avg=270.68, stdev=44.50
     lat (msec): min=183, max=1718, avg=270.68, stdev=44.50
    clat percentiles (msec):
     |  1.00th=[  199],  5.00th=[  209], 10.00th=[  220], 20.00th=[  243],
     | 30.00th=[  262], 40.00th=[  268], 50.00th=[  271], 60.00th=[  275],
     | 70.00th=[  279], 80.00th=[  284], 90.00th=[  305], 95.00th=[  342],
     | 99.00th=[  468], 99.50th=[  489], 99.90th=[  518], 99.95th=[  527],
     | 99.99th=[  718]
   bw (  KiB/s): min= 1960, max= 6112, per=6.25%, avg=3791.20, stdev=491.70, samples=1920
   iops        : min=  490, max= 1528, avg=947.79, stdev=122.92, samples=1920
  lat (msec)   : 250=23.64%, 500=76.50%, 750=0.30%, 1000=0.01%
  cpu          : usr=0.18%, sys=0.63%, ctx=805270, majf=0, minf=41
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=103.8%
     submit    : 0=0.0%, 4=99.5%, 8=0.4%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.1%
     complete  : 0=0.0%, 4=99.5%, 8=0.4%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.1%
     issued rwts: total=0,910000,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
  WRITE: bw=59.2MiB/s (62.1MB/s), 59.2MiB/s-59.2MiB/s (62.1MB/s-62.1MB/s), io=3571MiB (3744MB), run=60282-60282msec

随机读

fio –name=read_iops –directory=./oss-read –size=4G
–time_based –runtime=60s –ramp_time=2s –ioengine=libaio –direct=1
–verify=0 –bs=4K –iodepth=256 –rw=randread –group_reporting=1
–iodepth_batch_submit=256 –iodepth_batch_complete_max=256 –numjobs=16

job 1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
read_iops: (groupid=0, jobs=1): err= 0: pid=1659: Thu May 25 14:23:56 2023
   read: IOPS=14.9k, BW=58.0MiB/s (60.9MB/s)(3506MiB/60400msec)
    slat (nsec): min=857, max=3507.1k, avg=4631.18, stdev=21586.16
    clat (usec): min=549, max=9902.2k, avg=17228.91, stdev=33852.05
     lat (msec): min=2, max=9902, avg=17.23, stdev=33.85
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    4], 20.00th=[    4],
     | 30.00th=[    4], 40.00th=[    5], 50.00th=[    6], 60.00th=[   20],
     | 70.00th=[   22], 80.00th=[   23], 90.00th=[   44], 95.00th=[   52],
     | 99.00th=[  101], 99.50th=[  165], 99.90th=[  313], 99.95th=[  430],
     | 99.99th=[  919]
   bw (  KiB/s): min=58530, max=61464, per=100.00%, avg=59815.16, stdev=576.72, samples=120
   iops        : min=14632, max=15366, avg=14953.77, stdev=144.19, samples=120
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=38.16%, 10=16.75%, 20=7.54%, 50=31.87%
  lat (msec)   : 100=4.68%, 250=0.81%, 500=0.16%, 750=0.03%, 1000=0.01%
  cpu          : usr=1.70%, sys=9.08%, ctx=646895, majf=0, minf=4
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=105.0%
     submit    : 0=0.0%, 4=99.8%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.1%
     complete  : 0=0.0%, 4=99.8%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.1%, >=64=0.1%
     issued rwts: total=897333,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=58.0MiB/s (60.9MB/s), 58.0MiB/s-58.0MiB/s (60.9MB/s-60.9MB/s), io=3506MiB (3677MB), run=60400-60400msec

job 16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
read_iops: (groupid=0, jobs=16): err= 0: pid=1753: Thu May 25 14:31:45 2023
   read: IOPS=13.6k, BW=53.5MiB/s (56.1MB/s)(3521MiB/65783msec)
    slat (nsec): min=1039, max=1215.9k, avg=4108.54, stdev=6157.16
    clat (msec): min=101, max=17118, avg=299.33, stdev=378.46
     lat (msec): min=101, max=17118, avg=299.33, stdev=378.46
    clat percentiles (msec):
     |  1.00th=[  167],  5.00th=[  255], 10.00th=[  259], 20.00th=[  264],
     | 30.00th=[  266], 40.00th=[  271], 50.00th=[  271], 60.00th=[  275],
     | 70.00th=[  279], 80.00th=[  284], 90.00th=[  288], 95.00th=[  313],
     | 99.00th=[  414], 99.50th=[  531], 99.90th=[ 5940], 99.95th=[ 6007],
     | 99.99th=[ 6007]
   bw (  KiB/s): min=    8, max= 4744, per=6.77%, avg=3712.68, stdev=346.95, samples=1933
   iops        : min=    2, max= 1186, avg=928.17, stdev=86.74, samples=1933
  lat (msec)   : 250=3.56%, 500=96.38%, 750=0.05%, 1000=0.01%
  cpu          : usr=0.18%, sys=0.69%, ctx=915737, majf=0, minf=46
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=105.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.0%, >=64=0.1%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.1%, 32=0.1%, 64=0.0%, >=64=0.1%
     issued rwts: total=897169,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=53.5MiB/s (56.1MB/s), 53.5MiB/s-53.5MiB/s (56.1MB/s-56.1MB/s), io=3521MiB (3692MB), run=65783-65783msec

顺序读

fio –name=read_throughput –directory=./oss-read –numjobs=1
–size=4G –time_based –runtime=60s –ramp_time=2s –ioengine=libaio
–direct=1 –verify=0 –bs=1M –iodepth=64 –rw=read
–group_reporting=1
–iodepth_batch_submit=64 –iodepth_batch_complete_max=64

job 1 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
read_throughput: (groupid=0, jobs=1): err= 0: pid=1563: Thu May 25 14:17:27 2023
   read: IOPS=172, BW=174MiB/s (182MB/s)(10.6GiB/62624msec)
    slat (nsec): min=27575, max=95594, avg=44283.68, stdev=7853.96
    clat (msec): min=7, max=34280, avg=372.00, stdev=1503.11
     lat (msec): min=7, max=34280, avg=372.05, stdev=1503.11
    clat percentiles (msec):
     |  1.00th=[    9],  5.00th=[   10], 10.00th=[   11], 20.00th=[   13],
     | 30.00th=[   28], 40.00th=[   43], 50.00th=[   64], 60.00th=[  102],
     | 70.00th=[  163], 80.00th=[  262], 90.00th=[  542], 95.00th=[ 1167],
     | 99.00th=[ 7215], 99.50th=[11342], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min= 2048, max=292864, per=100.00%, avg=183271.33, stdev=39195.33, samples=121
   iops        : min=    2, max=  286, avg=178.95, stdev=38.27, samples=121
  lat (msec)   : 10=10.08%, 20=16.02%, 50=16.10%, 100=17.86%, 250=19.47%
  lat (msec)   : 500=9.76%, 750=3.96%, 1000=0.91%
  cpu          : usr=0.02%, sys=0.89%, ctx=10915, majf=0, minf=4
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=105.2%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=10829,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=174MiB/s (182MB/s), 174MiB/s-174MiB/s (182MB/s-182MB/s), io=10.6GiB (11.4GB), run=62624-62624msec

job 16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
read_throughput: (groupid=0, jobs=16): err= 0: pid=1517: Thu May 25 14:15:44 2023
   read: IOPS=151, BW=165MiB/s (173MB/s)(11.6GiB/72000msec)
    slat (usec): min=30, max=792, avg=51.00, stdev=12.83
    clat (msec): min=1951, max=60835, avg=5958.81, stdev=3116.79
     lat (msec): min=2010, max=60835, avg=5961.93, stdev=3115.09
    clat percentiles (msec):
     |  1.00th=[ 2601],  5.00th=[ 4329], 10.00th=[ 4396], 20.00th=[ 4530],
     | 30.00th=[ 4665], 40.00th=[ 4799], 50.00th=[ 4933], 60.00th=[ 5067],
     | 70.00th=[ 5470], 80.00th=[ 6074], 90.00th=[ 9194], 95.00th=[11879],
     | 99.00th=[17113], 99.50th=[17113], 99.90th=[17113], 99.95th=[17113],
     | 99.99th=[17113]
   bw (  KiB/s): min= 1885, max=67584, per=7.10%, avg=12009.61, stdev=6990.55, samples=1855
   iops        : min=    1, max=   66, avg=11.67, stdev= 6.83, samples=1855
  cpu          : usr=0.00%, sys=0.06%, ctx=11947, majf=0, minf=39
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=107.5%
     submit    : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     complete  : 0=0.0%, 4=99.9%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=10890,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=11.6GiB (12.5GB), run=72000-72000msec

原文作者：Momo

原文链接：https://mo.xmomo521.top/2023/05/25/对象存储-NAS-本地盘对比/

发表日期：May 25th 2023, 2:59:32 pm

更新日期：May 31st 2023, 2:27:46 pm

版权声明：本文采用知识共享署名-非商业性使用 4.0 国际许可协议进行许可

CATALOG
  1. 1. 前言
  2. 2. OSS 测试
    1. 2.1. 写入
    2. 2.2. 读取
  3. 3. ESSD磁盘
    1. 3.1. 顺序写入
    2. 3.2. 随机写
    3. 3.3. 顺序读
    4. 3.4. 随机读
  4. 4. 顺序读
  5. 5. NAS
    1. 5.1. 顺序写
    2. 5.2. 随机写 job1
    3. 5.3. 随机写 job 16
  6. 6. 随机读
  7. 7. 顺序读
Total : 133
2024
2023
2022
2021
2020
2019
Terraform CD TLS cfssl Python django drf JWT kubernetes CA 安全 Kibana JS Vue Devops Jenkins Gitlab DevOps Linux swarm pod ipvs ca k8s CPU CFS Linux时区 CST monitor PromQL Ajax logging Channels 聊天室 RPC Docker 学习笔记 Thanos Process ansible cryptography docker django-rest-framework Git 流程图 js 前端 Kubernetes prometheus-operator logstash filebeat Prometheus-operator eslint network axios vue cd xterm Docker网络 Prom联邦集群 secret CI ElementUi backup acme.sh Vuex Daemonset Taints log 动态路由 Swarm 软路由 标准运维 蓝鲸智云 ElementUI 监控 nginx ERROR
Ansible Atlantis Argocd Cloud-init Alpine Django CA Go etcd Elasticsearch Helm 日志 前端 Jenkins Kubernetes Jsonnet Docker kubernetes Linux Nginx NAS Grafana PostgreSQL Prometheus Python TerraForm Redis Terraform python alertmanager alpine docker Git gitlab-runner k3s ELK Mysql SSH vscode calico Python/Django LVM mysql Mongo TLS logging myself 蓝鲸 OpenWrt 源码分析 TCPDUMP Chrome Containerd iptables 技巧 nginx