s2p6-evo870-4k-sync-write-1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.12 Starting 1 process s2p6-evo870-4k-sync-write-1: (groupid=0, jobs=1): err= 0: pid=672929: Sat Apr 15 13:40:17 2023 write: IOPS=19.1k, BW=74.8MiB/s (78.4MB/s)(4487MiB/60001msec); 0 zone resets clat (usec): min=32, max=21511, avg=42.08, stdev=63.97 lat (usec): min=32, max=21511, avg=42.29, stdev=63.97 clat percentiles (usec): | 1.00th=[ 37], 5.00th=[ 37], 10.00th=[ 37], 20.00th=[ 37], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 38], 60.00th=[ 38], | 70.00th=[ 39], 80.00th=[ 41], 90.00th=[ 49], 95.00th=[ 63], | 99.00th=[ 78], 99.50th=[ 92], 99.90th=[ 611], 99.95th=[ 701], | 99.99th=[ 1549] bw ( KiB/s): min=69808, max=79840, per=99.99%, avg=76559.52, stdev=1618.50, samples=119 iops : min=17452, max=19960, avg=19139.86, stdev=404.59, samples=119 lat (usec) : 50=90.51%, 100=9.10%, 250=0.19%, 500=0.02%, 750=0.14% lat (usec) : 1000=0.02% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=352, avg= 7.85, stdev= 2.05 sync percentiles (nsec): | 1.00th=[ 7136], 5.00th=[ 7264], 10.00th=[ 7328], 20.00th=[ 7392], | 30.00th=[ 7456], 40.00th=[ 7456], 50.00th=[ 7520], 60.00th=[ 7520], | 70.00th=[ 7584], 80.00th=[ 7712], 90.00th=[ 8160], 95.00th=[ 9408], | 99.00th=[13632], 99.50th=[19072], 99.90th=[34048], 99.95th=[39168], | 99.99th=[55552] cpu : usr=8.45%, sys=15.47%, ctx=2297233, majf=0, minf=37 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1148578,0,1148577 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=74.8MiB/s (78.4MB/s), 74.8MiB/s-74.8MiB/s (78.4MB/s-78.4MB/s), io=4487MiB (4705MB), run=60001-60001msec Disk stats (read/write): sdf: ios=22/2292852, merge=0/0, ticks=6/48148, in_queue=40872, util=68.16% s2p6-evo870-4k-sync-write-2: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 2 processes s2p6-evo870-4k-sync-write-2: (groupid=0, jobs=2): err= 0: pid=675395: Sat Apr 15 13:41:17 2023 write: IOPS=27.4k, BW=107MiB/s (112MB/s)(6414MiB/60001msec); 0 zone resets clat (usec): min=32, max=24739, avg=54.62, stdev=66.12 lat (usec): min=32, max=24740, avg=54.84, stdev=66.12 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 46], | 30.00th=[ 49], 40.00th=[ 50], 50.00th=[ 51], 60.00th=[ 51], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 63], 95.00th=[ 79], | 99.00th=[ 109], 99.50th=[ 121], 99.90th=[ 635], 99.95th=[ 824], | 99.99th=[ 1942] bw ( KiB/s): min=49536, max=58048, per=50.01%, avg=54749.67, stdev=1693.74, samples=239 iops : min=12384, max=14512, avg=13687.42, stdev=423.43, samples=239 lat (usec) : 50=45.20%, 100=53.07%, 250=1.49%, 500=0.04%, 750=0.15% lat (usec) : 1000=0.02% lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=823, avg=16.19, stdev= 5.47 sync percentiles (usec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15], | 70.00th=[ 16], 80.00th=[ 18], 90.00th=[ 25], 95.00th=[ 27], | 99.00th=[ 34], 99.50th=[ 39], 99.90th=[ 53], 99.95th=[ 61], | 99.99th=[ 108] cpu : usr=4.89%, sys=19.45%, ctx=3287748, majf=0, minf=88 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1642055,0,1642053 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=107MiB/s (112MB/s), 107MiB/s-107MiB/s (112MB/s-112MB/s), io=6414MiB (6726MB), run=60001-60001msec Disk stats (read/write): sdf: ios=105/3278231, merge=0/6, ticks=15/94442, in_queue=76364, util=80.65% s2p6-evo870-4k-sync-write-3: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 3 processes s2p6-evo870-4k-sync-write-3: (groupid=0, jobs=3): err= 0: pid=677961: Sat Apr 15 13:42:18 2023 write: IOPS=38.5k, BW=150MiB/s (158MB/s)(9027MiB/60001msec); 0 zone resets clat (usec): min=34, max=18305, avg=56.88, stdev=74.99 lat (usec): min=35, max=18306, avg=57.09, stdev=75.00 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 43], 20.00th=[ 45], | 30.00th=[ 47], 40.00th=[ 49], 50.00th=[ 51], 60.00th=[ 52], | 70.00th=[ 55], 80.00th=[ 58], 90.00th=[ 67], 95.00th=[ 88], | 99.00th=[ 123], 99.50th=[ 192], 99.90th=[ 865], 99.95th=[ 1254], | 99.99th=[ 2507] bw ( KiB/s): min=45880, max=54440, per=33.33%, avg=51350.25, stdev=1538.79, samples=360 iops : min=11470, max=13610, avg=12837.53, stdev=384.68, samples=360 lat (usec) : 50=48.52%, 100=48.21%, 250=2.86%, 500=0.11%, 750=0.16% lat (usec) : 1000=0.05% lat (msec) : 2=0.06%, 4=0.01%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=1070, avg=18.62, stdev= 7.59 sync percentiles (usec): | 1.00th=[ 10], 5.00th=[ 12], 10.00th=[ 13], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 16], 60.00th=[ 18], | 70.00th=[ 20], 80.00th=[ 24], 90.00th=[ 30], 95.00th=[ 33], | 99.00th=[ 43], 99.50th=[ 48], 99.90th=[ 65], 99.95th=[ 75], | 99.99th=[ 120] cpu : usr=4.89%, sys=19.49%, ctx=4648890, majf=0, minf=91 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2310960,0,2310957 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=150MiB/s (158MB/s), 150MiB/s-150MiB/s (158MB/s-158MB/s), io=9027MiB (9466MB), run=60001-60001msec Disk stats (read/write): sdf: ios=105/4613353, merge=0/800, ticks=11/139218, in_queue=105580, util=93.92% s2p6-evo870-4k-sync-write-4: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 4 processes s2p6-evo870-4k-sync-write-4: (groupid=0, jobs=4): err= 0: pid=681006: Sat Apr 15 13:43:19 2023 write: IOPS=47.5k, BW=186MiB/s (195MB/s)(10.9GiB/60001msec); 0 zone resets clat (usec): min=35, max=23046, avg=60.82, stdev=97.70 lat (usec): min=35, max=23047, avg=61.02, stdev=97.71 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 44], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 51], 50.00th=[ 53], 60.00th=[ 56], | 70.00th=[ 59], 80.00th=[ 62], 90.00th=[ 71], 95.00th=[ 91], | 99.00th=[ 145], 99.50th=[ 249], 99.90th=[ 988], 99.95th=[ 1369], | 99.99th=[ 2737] bw ( KiB/s): min=34536, max=51456, per=25.01%, avg=47559.57, stdev=2180.42, samples=476 iops : min= 8634, max=12864, avg=11889.88, stdev=545.11, samples=476 lat (usec) : 50=33.83%, 100=62.51%, 250=3.16%, 500=0.16%, 750=0.17% lat (usec) : 1000=0.07% lat (msec) : 2=0.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=1918, avg=21.02, stdev= 9.34 sync percentiles (usec): | 1.00th=[ 11], 5.00th=[ 13], 10.00th=[ 13], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 17], 50.00th=[ 18], 60.00th=[ 21], | 70.00th=[ 24], 80.00th=[ 29], 90.00th=[ 33], 95.00th=[ 38], | 99.00th=[ 50], 99.50th=[ 57], 99.90th=[ 76], 99.95th=[ 87], | 99.99th=[ 143] cpu : usr=4.40%, sys=18.37%, ctx=5778025, majf=0, minf=99 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2851999,0,2851997 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=186MiB/s (195MB/s), 186MiB/s-186MiB/s (195MB/s-195MB/s), io=10.9GiB (11.7GB), run=60001-60001msec Disk stats (read/write): sdf: ios=103/5693671, merge=0/60, ticks=11/184699, in_queue=136824, util=98.54% s2p6-evo870-4k-sync-write-5: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 5 processes s2p6-evo870-4k-sync-write-5: (groupid=0, jobs=5): err= 0: pid=683564: Sat Apr 15 13:44:20 2023 write: IOPS=52.5k, BW=205MiB/s (215MB/s)(12.0GiB/60001msec); 0 zone resets clat (usec): min=38, max=39450, avg=66.01, stdev=97.86 lat (usec): min=38, max=39450, avg=66.23, stdev=97.86 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 54], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 78], 95.00th=[ 98], | 99.00th=[ 163], 99.50th=[ 269], 99.90th=[ 1020], 99.95th=[ 1434], | 99.99th=[ 2835] bw ( KiB/s): min=35376, max=45560, per=20.00%, avg=41974.07, stdev=1934.86, samples=596 iops : min= 8844, max=11390, avg=10493.49, stdev=483.73, samples=596 lat (usec) : 50=15.85%, 100=79.41%, 250=4.20%, 500=0.19%, 750=0.17% lat (usec) : 1000=0.08% lat (msec) : 2=0.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=3991, avg=26.86, stdev=14.70 sync percentiles (usec): | 1.00th=[ 12], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 28], | 70.00th=[ 32], 80.00th=[ 36], 90.00th=[ 42], 95.00th=[ 47], | 99.00th=[ 63], 99.50th=[ 74], 99.90th=[ 122], 99.95th=[ 169], | 99.99th=[ 302] cpu : usr=4.01%, sys=17.65%, ctx=6497894, majf=0, minf=165 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3148763,0,3148759 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=12.0GiB (12.9GB), run=60001-60001msec Disk stats (read/write): sdf: ios=108/6284880, merge=0/476, ticks=12/225889, in_queue=158564, util=98.97% s2p6-evo870-4k-sync-write-6: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 6 processes s2p6-evo870-4k-sync-write-6: (groupid=0, jobs=6): err= 0: pid=686078: Sat Apr 15 13:45:20 2023 write: IOPS=56.5k, BW=221MiB/s (231MB/s)(12.9GiB/60001msec); 0 zone resets clat (usec): min=37, max=30250, avg=70.03, stdev=98.66 lat (usec): min=38, max=30251, avg=70.24, stdev=98.66 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 49], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 59], 50.00th=[ 62], 60.00th=[ 65], | 70.00th=[ 69], 80.00th=[ 74], 90.00th=[ 84], 95.00th=[ 104], | 99.00th=[ 180], 99.50th=[ 285], 99.90th=[ 1004], 99.95th=[ 1418], | 99.99th=[ 2638] bw ( KiB/s): min=31584, max=40560, per=16.66%, avg=37654.66, stdev=1459.32, samples=714 iops : min= 7896, max=10140, avg=9413.63, stdev=364.82, samples=714 lat (usec) : 50=8.38%, 100=86.23%, 250=4.83%, 500=0.21%, 750=0.17% lat (usec) : 1000=0.09% lat (msec) : 2=0.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=5386, avg=33.91, stdev=18.57 sync percentiles (usec): | 1.00th=[ 14], 5.00th=[ 16], 10.00th=[ 18], 20.00th=[ 21], | 30.00th=[ 26], 40.00th=[ 30], 50.00th=[ 33], 60.00th=[ 37], | 70.00th=[ 40], 80.00th=[ 45], 90.00th=[ 51], 95.00th=[ 58], | 99.00th=[ 75], 99.50th=[ 85], 99.90th=[ 128], 99.95th=[ 172], | 99.99th=[ 367] cpu : usr=3.66%, sys=17.12%, ctx=7204549, majf=0, minf=208 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3389827,0,3389823 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=221MiB/s (231MB/s), 221MiB/s-221MiB/s (231MB/s-231MB/s), io=12.9GiB (13.9GB), run=60001-60001msec Disk stats (read/write): sdf: ios=102/6765605, merge=0/628, ticks=12/272383, in_queue=181944, util=99.63% s2p6-evo870-4k-sync-write-7: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 7 processes s2p6-evo870-4k-sync-write-7: (groupid=0, jobs=7): err= 0: pid=688632: Sat Apr 15 13:46:21 2023 write: IOPS=60.3k, BW=235MiB/s (247MB/s)(13.8GiB/60001msec); 0 zone resets clat (usec): min=38, max=35192, avg=73.06, stdev=101.33 lat (usec): min=39, max=35192, avg=73.29, stdev=101.34 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 66], 60.00th=[ 70], | 70.00th=[ 73], 80.00th=[ 79], 90.00th=[ 90], 95.00th=[ 110], | 99.00th=[ 174], 99.50th=[ 208], 99.90th=[ 791], 99.95th=[ 1172], | 99.99th=[ 2671] bw ( KiB/s): min=30440, max=37064, per=14.28%, avg=34437.97, stdev=1116.52, samples=835 iops : min= 7610, max= 9266, avg=8609.46, stdev=279.14, samples=835 lat (usec) : 50=4.89%, 100=88.74%, 250=5.98%, 500=0.12%, 750=0.14% lat (usec) : 1000=0.06% lat (msec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=4965, avg=40.71, stdev=17.69 sync percentiles (usec): | 1.00th=[ 15], 5.00th=[ 19], 10.00th=[ 22], 20.00th=[ 29], | 30.00th=[ 33], 40.00th=[ 37], 50.00th=[ 40], 60.00th=[ 43], | 70.00th=[ 47], 80.00th=[ 52], 90.00th=[ 59], 95.00th=[ 67], | 99.00th=[ 87], 99.50th=[ 98], 99.90th=[ 139], 99.95th=[ 174], | 99.99th=[ 338] cpu : usr=3.40%, sys=17.48%, ctx=7866391, majf=0, minf=250 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3616454,0,3616450 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=235MiB/s (247MB/s), 235MiB/s-235MiB/s (247MB/s-247MB/s), io=13.8GiB (14.8GB), run=60001-60001msec Disk stats (read/write): sdf: ios=105/7217768, merge=0/659, ticks=14/316775, in_queue=200068, util=99.73% s2p6-evo870-4k-sync-write-8: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 8 processes s2p6-evo870-4k-sync-write-8: (groupid=0, jobs=8): err= 0: pid=691574: Sat Apr 15 13:47:22 2023 write: IOPS=63.6k, BW=248MiB/s (260MB/s)(14.6GiB/60001msec); 0 zone resets clat (usec): min=40, max=49399, avg=78.73, stdev=119.86 lat (usec): min=40, max=49400, avg=78.94, stdev=119.86 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 53], 10.00th=[ 56], 20.00th=[ 60], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 70], 60.00th=[ 74], | 70.00th=[ 78], 80.00th=[ 85], 90.00th=[ 97], 95.00th=[ 120], | 99.00th=[ 200], 99.50th=[ 277], 99.90th=[ 1004], 99.95th=[ 1401], | 99.99th=[ 2704] bw ( KiB/s): min=22528, max=34576, per=12.50%, avg=31783.93, stdev=1324.80, samples=952 iops : min= 5632, max= 8644, avg=7945.96, stdev=331.20, samples=952 lat (usec) : 50=2.79%, 100=88.44%, 250=8.19%, 500=0.25%, 750=0.15% lat (usec) : 1000=0.09% lat (msec) : 2=0.08%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=2841, avg=44.82, stdev=19.61 sync percentiles (usec): | 1.00th=[ 16], 5.00th=[ 21], 10.00th=[ 27], 20.00th=[ 33], | 30.00th=[ 37], 40.00th=[ 40], 50.00th=[ 43], 60.00th=[ 46], | 70.00th=[ 51], 80.00th=[ 56], 90.00th=[ 65], 95.00th=[ 75], | 99.00th=[ 98], 99.50th=[ 110], 99.90th=[ 157], 99.95th=[ 206], | 99.99th=[ 498] cpu : usr=3.07%, sys=17.20%, ctx=8365383, majf=0, minf=253 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3815165,0,3815160 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=248MiB/s (260MB/s), 248MiB/s-248MiB/s (260MB/s-260MB/s), io=14.6GiB (15.6GB), run=60001-60001msec Disk stats (read/write): sdf: ios=105/7612892, merge=0/1888, ticks=12/367281, in_queue=226092, util=99.50% s2p6-evo870-4k-sync-write-9: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 9 processes s2p6-evo870-4k-sync-write-9: (groupid=0, jobs=9): err= 0: pid=694029: Sat Apr 15 13:48:22 2023 write: IOPS=66.7k, BW=261MiB/s (273MB/s)(15.3GiB/60008msec); 0 zone resets clat (usec): min=40, max=19610, avg=83.87, stdev=98.79 lat (usec): min=40, max=19611, avg=84.10, stdev=98.80 clat percentiles (usec): | 1.00th=[ 50], 5.00th=[ 56], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 83], 80.00th=[ 90], 90.00th=[ 104], 95.00th=[ 127], | 99.00th=[ 219], 99.50th=[ 318], 99.90th=[ 1090], 99.95th=[ 1467], | 99.99th=[ 2835] bw ( KiB/s): min=22048, max=32432, per=11.11%, avg=29659.41, stdev=1267.21, samples=1076 iops : min= 5512, max= 8108, avg=7414.83, stdev=316.80, samples=1076 lat (usec) : 50=1.31%, 100=86.87%, 250=11.10%, 500=0.35%, 750=0.14% lat (usec) : 1000=0.12% lat (msec) : 2=0.10%, 4=0.02%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=3397, avg=48.54, stdev=23.28 sync percentiles (usec): | 1.00th=[ 17], 5.00th=[ 24], 10.00th=[ 31], 20.00th=[ 36], | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 49], | 70.00th=[ 54], 80.00th=[ 60], 90.00th=[ 71], 95.00th=[ 82], | 99.00th=[ 109], 99.50th=[ 122], 99.90th=[ 174], 99.95th=[ 235], | 99.99th=[ 652] cpu : usr=3.12%, sys=16.94%, ctx=8879511, majf=0, minf=288 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4004812,0,4004804 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=261MiB/s (273MB/s), 261MiB/s-261MiB/s (273MB/s-273MB/s), io=15.3GiB (16.4GB), run=60008-60008msec Disk stats (read/write): sdf: ios=102/7992795, merge=0/1471, ticks=13/416899, in_queue=255276, util=99.57% s2p6-evo870-4k-sync-write-10: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 10 processes s2p6-evo870-4k-sync-write-10: (groupid=0, jobs=10): err= 0: pid=696650: Sat Apr 15 13:49:23 2023 write: IOPS=70.1k, BW=274MiB/s (287MB/s)(16.0GiB/60001msec); 0 zone resets clat (usec): min=40, max=25990, avg=90.72, stdev=124.86 lat (usec): min=41, max=25990, avg=90.94, stdev=124.86 clat percentiles (usec): | 1.00th=[ 53], 5.00th=[ 60], 10.00th=[ 64], 20.00th=[ 70], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 90], 80.00th=[ 97], 90.00th=[ 111], 95.00th=[ 133], | 99.00th=[ 239], 99.50th=[ 396], 99.90th=[ 1188], 99.95th=[ 1614], | 99.99th=[ 2966] bw ( KiB/s): min=23176, max=30992, per=10.00%, avg=28023.69, stdev=1049.01, samples=1191 iops : min= 5794, max= 7748, avg=7005.90, stdev=262.25, samples=1191 lat (usec) : 50=0.48%, 100=82.89%, 250=15.74%, 500=0.46%, 750=0.13% lat (usec) : 1000=0.16% lat (msec) : 2=0.11%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=8, max=2762, avg=49.52, stdev=22.65 sync percentiles (usec): | 1.00th=[ 18], 5.00th=[ 26], 10.00th=[ 31], 20.00th=[ 36], | 30.00th=[ 40], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 50], | 70.00th=[ 55], 80.00th=[ 61], 90.00th=[ 74], 95.00th=[ 86], | 99.00th=[ 116], 99.50th=[ 131], 99.90th=[ 202], 99.95th=[ 265], | 99.99th=[ 603] cpu : usr=2.88%, sys=16.38%, ctx=9380983, majf=0, minf=290 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4205439,0,4205432 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=274MiB/s (287MB/s), 274MiB/s-274MiB/s (287MB/s-287MB/s), io=16.0GiB (17.2GB), run=60001-60001msec Disk stats (read/write): sdf: ios=105/8392008, merge=0/2034, ticks=12/470968, in_queue=295236, util=99.46% s2p6-evo870-4k-sync-write-11: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 11 processes s2p6-evo870-4k-sync-write-11: (groupid=0, jobs=11): err= 0: pid=699766: Sat Apr 15 13:50:24 2023 write: IOPS=71.1k, BW=278MiB/s (291MB/s)(16.3GiB/60001msec); 0 zone resets clat (usec): min=38, max=18673, avg=98.81, stdev=97.17 lat (usec): min=38, max=18673, avg=99.04, stdev=97.18 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 64], 10.00th=[ 69], 20.00th=[ 75], | 30.00th=[ 79], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 91], | 70.00th=[ 97], 80.00th=[ 105], 90.00th=[ 122], 95.00th=[ 151], | 99.00th=[ 310], 99.50th=[ 635], 99.90th=[ 1205], 99.95th=[ 1532], | 99.99th=[ 2704] bw ( KiB/s): min=21800, max=28480, per=9.09%, avg=25846.09, stdev=1137.02, samples=1310 iops : min= 5450, max= 7120, avg=6461.50, stdev=284.25, samples=1310 lat (usec) : 50=0.22%, 100=74.34%, 250=23.93%, 500=0.92%, 750=0.18% lat (usec) : 1000=0.24% lat (msec) : 2=0.14%, 4=0.01%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=14467, avg=53.64, stdev=35.89 sync percentiles (usec): | 1.00th=[ 19], 5.00th=[ 29], 10.00th=[ 34], 20.00th=[ 38], | 30.00th=[ 42], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 53], | 70.00th=[ 59], 80.00th=[ 67], 90.00th=[ 81], 95.00th=[ 95], | 99.00th=[ 129], 99.50th=[ 149], 99.90th=[ 227], 99.95th=[ 318], | 99.99th=[ 947] cpu : usr=2.47%, sys=16.58%, ctx=9538088, majf=0, minf=374 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4265150,0,4265139 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=278MiB/s (291MB/s), 278MiB/s-278MiB/s (291MB/s-291MB/s), io=16.3GiB (17.5GB), run=60001-60001msec Disk stats (read/write): sdf: ios=105/8510903, merge=0/1478, ticks=13/520852, in_queue=323572, util=99.06% s2p6-evo870-4k-sync-write-12: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 12 processes s2p6-evo870-4k-sync-write-12: (groupid=0, jobs=12): err= 0: pid=702436: Sat Apr 15 13:51:25 2023 write: IOPS=73.0k, BW=289MiB/s (303MB/s)(16.9GiB/60001msec); 0 zone resets clat (usec): min=37, max=24830, avg=104.08, stdev=88.50 lat (usec): min=37, max=24830, avg=104.31, stdev=88.51 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 68], 10.00th=[ 74], 20.00th=[ 80], | 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 94], 60.00th=[ 99], | 70.00th=[ 105], 80.00th=[ 114], 90.00th=[ 130], 95.00th=[ 155], | 99.00th=[ 277], 99.50th=[ 465], 99.90th=[ 1057], 99.95th=[ 1336], | 99.99th=[ 2180] bw ( KiB/s): min=18784, max=27320, per=8.33%, avg=24661.41, stdev=1137.54, samples=1428 iops : min= 4696, max= 6830, avg=6165.34, stdev=284.39, samples=1428 lat (usec) : 50=0.14%, 100=62.22%, 250=36.34%, 500=0.82%, 750=0.13% lat (usec) : 1000=0.22% lat (msec) : 2=0.11%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=6094, avg=55.84, stdev=28.10 sync percentiles (usec): | 1.00th=[ 19], 5.00th=[ 30], 10.00th=[ 34], 20.00th=[ 39], | 30.00th=[ 43], 40.00th=[ 46], 50.00th=[ 50], 60.00th=[ 55], | 70.00th=[ 61], 80.00th=[ 71], 90.00th=[ 86], 95.00th=[ 101], | 99.00th=[ 139], 99.50th=[ 155], 99.90th=[ 217], 99.95th=[ 289], | 99.99th=[ 799] cpu : usr=2.34%, sys=16.69%, ctx=10016659, majf=0, minf=396 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4439446,0,4439439 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=289MiB/s (303MB/s), 289MiB/s-289MiB/s (303MB/s-303MB/s), io=16.9GiB (18.2GB), run=60001-60001msec Disk stats (read/write): sdf: ios=102/8856851, merge=0/3034, ticks=13/575296, in_queue=357304, util=99.05% s2p6-evo870-4k-sync-write-13: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 13 processes s2p6-evo870-4k-sync-write-13: (groupid=0, jobs=13): err= 0: pid=704911: Sat Apr 15 13:52:25 2023 write: IOPS=73.6k, BW=287MiB/s (301MB/s)(16.8GiB/60001msec); 0 zone resets clat (usec): min=38, max=24535, avg=112.96, stdev=115.18 lat (usec): min=38, max=24535, avg=113.19, stdev=115.18 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 73], 10.00th=[ 79], 20.00th=[ 87], | 30.00th=[ 93], 40.00th=[ 98], 50.00th=[ 102], 60.00th=[ 109], | 70.00th=[ 115], 80.00th=[ 124], 90.00th=[ 143], 95.00th=[ 172], | 99.00th=[ 269], 99.50th=[ 404], 99.90th=[ 1221], 99.95th=[ 1647], | 99.99th=[ 2999] bw ( KiB/s): min=20792, max=24608, per=7.69%, avg=22635.76, stdev=652.80, samples=1547 iops : min= 5198, max= 6152, avg=5658.91, stdev=163.19, samples=1547 lat (usec) : 50=0.08%, 100=45.42%, 250=53.21%, 500=0.86%, 750=0.09% lat (usec) : 1000=0.18% lat (msec) : 2=0.12%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=7099, avg=61.25, stdev=33.81 sync percentiles (usec): | 1.00th=[ 21], 5.00th=[ 32], 10.00th=[ 37], 20.00th=[ 42], | 30.00th=[ 46], 40.00th=[ 50], 50.00th=[ 55], 60.00th=[ 60], | 70.00th=[ 68], 80.00th=[ 78], 90.00th=[ 96], 95.00th=[ 115], | 99.00th=[ 157], 99.50th=[ 178], 99.90th=[ 247], 99.95th=[ 326], | 99.99th=[ 898] cpu : usr=2.35%, sys=17.16%, ctx=10081477, majf=0, minf=402 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4414245,0,4414238 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=287MiB/s (301MB/s), 287MiB/s-287MiB/s (301MB/s-301MB/s), io=16.8GiB (18.1GB), run=60001-60001msec Disk stats (read/write): sdf: ios=105/8806644, merge=0/2881, ticks=12/624812, in_queue=382904, util=98.54% s2p6-evo870-4k-sync-write-14: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 14 processes s2p6-evo870-4k-sync-write-14: (groupid=0, jobs=14): err= 0: pid=707841: Sat Apr 15 13:53:26 2023 write: IOPS=72.0k, BW=285MiB/s (299MB/s)(16.7GiB/60001msec); 0 zone resets clat (usec): min=41, max=21157, avg=122.17, stdev=117.34 lat (usec): min=41, max=21157, avg=122.40, stdev=117.35 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 77], 10.00th=[ 85], 20.00th=[ 93], | 30.00th=[ 100], 40.00th=[ 105], 50.00th=[ 112], 60.00th=[ 118], | 70.00th=[ 125], 80.00th=[ 137], 90.00th=[ 159], 95.00th=[ 192], | 99.00th=[ 277], 99.50th=[ 379], 99.90th=[ 1090], 99.95th=[ 1500], | 99.99th=[ 3228] bw ( KiB/s): min=15296, max=22776, per=7.14%, avg=20844.83, stdev=778.80, samples=1666 iops : min= 3824, max= 5694, avg=5211.19, stdev=194.70, samples=1666 lat (usec) : 50=0.07%, 100=30.42%, 250=67.97%, 500=1.17%, 750=0.09% lat (usec) : 1000=0.16% lat (msec) : 2=0.10%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=4872, avg=67.21, stdev=38.50 sync percentiles (usec): | 1.00th=[ 22], 5.00th=[ 34], 10.00th=[ 39], 20.00th=[ 44], | 30.00th=[ 48], 40.00th=[ 53], 50.00th=[ 58], 60.00th=[ 65], | 70.00th=[ 74], 80.00th=[ 87], 90.00th=[ 110], 95.00th=[ 131], | 99.00th=[ 182], 99.50th=[ 204], 99.90th=[ 285], 99.95th=[ 379], | 99.99th=[ 1020] cpu : usr=2.11%, sys=17.98%, ctx=10119510, majf=0, minf=410 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4378230,0,4378222 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=285MiB/s (299MB/s), 285MiB/s-285MiB/s (299MB/s-299MB/s), io=16.7GiB (17.9GB), run=60001-60001msec Disk stats (read/write): sdf: ios=102/8735106, merge=0/3722, ticks=12/676584, in_queue=407328, util=97.85% s2p6-evo870-4k-sync-write-15: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 15 processes s2p6-evo870-4k-sync-write-15: (groupid=0, jobs=15): err= 0: pid=710516: Sat Apr 15 13:54:27 2023 write: IOPS=72.1k, BW=282MiB/s (295MB/s)(16.5GiB/60001msec); 0 zone resets clat (usec): min=39, max=15056, avg=132.29, stdev=95.63 lat (usec): min=40, max=15056, avg=132.54, stdev=95.64 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 81], 10.00th=[ 89], 20.00th=[ 99], | 30.00th=[ 106], 40.00th=[ 114], 50.00th=[ 120], 60.00th=[ 126], | 70.00th=[ 135], 80.00th=[ 147], 90.00th=[ 176], 95.00th=[ 212], | 99.00th=[ 330], 99.50th=[ 635], 99.90th=[ 1270], 99.95th=[ 1565], | 99.99th=[ 2606] bw ( KiB/s): min=17736, max=21360, per=6.67%, avg=19234.95, stdev=535.34, samples=1786 iops : min= 4434, max= 5340, avg=4808.72, stdev=133.84, samples=1786 lat (usec) : 50=0.05%, 100=21.14%, 250=76.18%, 500=2.06%, 750=0.14% lat (usec) : 1000=0.24% lat (msec) : 2=0.17%, 4=0.02%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=11487, avg=73.27, stdev=43.02 sync percentiles (usec): | 1.00th=[ 23], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45], | 30.00th=[ 50], 40.00th=[ 56], 50.00th=[ 62], 60.00th=[ 71], | 70.00th=[ 82], 80.00th=[ 97], 90.00th=[ 125], 95.00th=[ 149], | 99.00th=[ 206], 99.50th=[ 229], 99.90th=[ 310], 99.95th=[ 400], | 99.99th=[ 979] cpu : usr=1.86%, sys=18.83%, ctx=10115638, majf=0, minf=468 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4328409,0,4328399 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=282MiB/s (295MB/s), 282MiB/s-282MiB/s (295MB/s-295MB/s), io=16.5GiB (17.7GB), run=60001-60001msec Disk stats (read/write): sdf: ios=105/8631852, merge=0/5857, ticks=16/722999, in_queue=421484, util=96.54% s2p6-evo870-4k-sync-write-16: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 16 processes s2p6-evo870-4k-sync-write-16: (groupid=0, jobs=16): err= 0: pid=712993: Sat Apr 15 13:55:28 2023 write: IOPS=70.6k, BW=276MiB/s (289MB/s)(16.2GiB/60001msec); 0 zone resets clat (usec): min=40, max=14178, avg=144.23, stdev=84.40 lat (usec): min=41, max=14178, avg=144.47, stdev=84.41 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 86], 10.00th=[ 95], 20.00th=[ 108], | 30.00th=[ 116], 40.00th=[ 123], 50.00th=[ 130], 60.00th=[ 139], | 70.00th=[ 149], 80.00th=[ 163], 90.00th=[ 196], 95.00th=[ 237], | 99.00th=[ 388], 99.50th=[ 742], 99.90th=[ 1237], 99.95th=[ 1434], | 99.99th=[ 1827] bw ( KiB/s): min=15784, max=19984, per=6.25%, avg=17657.95, stdev=464.43, samples=1905 iops : min= 3946, max= 4996, avg=4414.48, stdev=116.11, samples=1905 lat (usec) : 50=0.05%, 100=13.78%, 250=82.11%, 500=3.36%, 750=0.22% lat (usec) : 1000=0.31% lat (msec) : 2=0.18%, 4=0.01%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=4212, avg=79.78, stdev=48.36 sync percentiles (usec): | 1.00th=[ 23], 5.00th=[ 35], 10.00th=[ 40], 20.00th=[ 46], | 30.00th=[ 52], 40.00th=[ 59], 50.00th=[ 67], 60.00th=[ 77], | 70.00th=[ 90], 80.00th=[ 110], 90.00th=[ 141], 95.00th=[ 167], | 99.00th=[ 229], 99.50th=[ 255], 99.90th=[ 334], 99.95th=[ 433], | 99.99th=[ 1254] cpu : usr=1.72%, sys=19.47%, ctx=10029111, majf=0, minf=588 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4238982,0,4238972 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=16.2GiB (17.4GB), run=60001-60001msec Disk stats (read/write): sdf: ios=102/8454761, merge=0/5058, ticks=13/774178, in_queue=451360, util=96.02%