s2p6-mx500-4k-sync-write-1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 fio-3.12 Starting 1 process s2p6-mx500-4k-sync-write-1: (groupid=0, jobs=1): err= 0: pid=755021: Sat Apr 15 14:11:27 2023 write: IOPS=18.5k, BW=72.4MiB/s (75.9MB/s)(4342MiB/60001msec); 0 zone resets clat (usec): min=31, max=23756, avg=43.33, stdev=78.60 lat (usec): min=31, max=23756, avg=43.53, stdev=78.60 clat percentiles (usec): | 1.00th=[ 34], 5.00th=[ 36], 10.00th=[ 37], 20.00th=[ 38], | 30.00th=[ 38], 40.00th=[ 38], 50.00th=[ 39], 60.00th=[ 39], | 70.00th=[ 40], 80.00th=[ 42], 90.00th=[ 51], 95.00th=[ 63], | 99.00th=[ 85], 99.50th=[ 118], 99.90th=[ 660], 99.95th=[ 840], | 99.99th=[ 2089] bw ( KiB/s): min=53704, max=82520, per=99.98%, avg=74089.93, stdev=3296.12, samples=119 iops : min=13426, max=20630, avg=18522.46, stdev=824.04, samples=119 lat (usec) : 50=89.66%, 100=9.69%, 250=0.37%, 500=0.05%, 750=0.15% lat (usec) : 1000=0.03% lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=844, avg= 8.36, stdev= 2.55 sync percentiles (nsec): | 1.00th=[ 7200], 5.00th=[ 7328], 10.00th=[ 7392], 20.00th=[ 7456], | 30.00th=[ 7520], 40.00th=[ 7584], 50.00th=[ 7584], 60.00th=[ 7648], | 70.00th=[ 7840], 80.00th=[ 8384], 90.00th=[10688], 95.00th=[11072], | 99.00th=[16192], 99.50th=[21376], 99.90th=[36096], 99.95th=[41216], | 99.99th=[56576] cpu : usr=7.29%, sys=16.67%, ctx=2223353, majf=0, minf=28 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1111645,0,1111644 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=72.4MiB/s (75.9MB/s), 72.4MiB/s-72.4MiB/s (75.9MB/s-75.9MB/s), io=4342MiB (4553MB), run=60001-60001msec Disk stats (read/write): sdg: ios=17/2218550, merge=0/0, ticks=1/48330, in_queue=40188, util=67.04% s2p6-mx500-4k-sync-write-2: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 2 processes s2p6-mx500-4k-sync-write-2: (groupid=0, jobs=2): err= 0: pid=757505: Sat Apr 15 14:12:28 2023 write: IOPS=26.9k, BW=105MiB/s (110MB/s)(6294MiB/60001msec); 0 zone resets clat (usec): min=33, max=17538, avg=56.10, stdev=76.97 lat (usec): min=33, max=17538, avg=56.30, stdev=76.97 clat percentiles (usec): | 1.00th=[ 40], 5.00th=[ 41], 10.00th=[ 41], 20.00th=[ 42], | 30.00th=[ 43], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 51], | 70.00th=[ 52], 80.00th=[ 56], 90.00th=[ 69], 95.00th=[ 90], | 99.00th=[ 174], 99.50th=[ 302], 99.90th=[ 881], 99.95th=[ 1188], | 99.99th=[ 2409] bw ( KiB/s): min=41864, max=63360, per=50.00%, avg=53707.16, stdev=4399.98, samples=238 iops : min=10466, max=15840, avg=13426.79, stdev=1100.02, samples=238 lat (usec) : 50=57.95%, 100=38.44%, 250=3.03%, 500=0.24%, 750=0.19% lat (usec) : 1000=0.08% lat (msec) : 2=0.06%, 4=0.01%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=654, avg=16.20, stdev= 6.68 sync percentiles (nsec): | 1.00th=[ 9152], 5.00th=[ 9536], 10.00th=[ 9792], 20.00th=[10432], | 30.00th=[11712], 40.00th=[13632], 50.00th=[14528], 60.00th=[14912], | 70.00th=[17280], 80.00th=[23936], 90.00th=[25472], 95.00th=[27520], | 99.00th=[34048], 99.50th=[38656], 99.90th=[53504], 99.95th=[60672], | 99.99th=[95744] cpu : usr=4.76%, sys=17.63%, ctx=3227789, majf=0, minf=66 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,1611390,0,1611389 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=105MiB/s (110MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=6294MiB (6600MB), run=60001-60001msec Disk stats (read/write): sdg: ios=108/3216197, merge=0/41, ticks=41/94498, in_queue=76408, util=81.20% s2p6-mx500-4k-sync-write-3: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 3 processes s2p6-mx500-4k-sync-write-3: (groupid=0, jobs=3): err= 0: pid=760519: Sat Apr 15 14:13:29 2023 write: IOPS=36.2k, BW=142MiB/s (148MB/s)(8494MiB/60001msec); 0 zone resets clat (usec): min=34, max=22796, avg=61.68, stdev=96.50 lat (usec): min=34, max=22796, avg=61.89, stdev=96.50 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 43], 10.00th=[ 44], 20.00th=[ 45], | 30.00th=[ 48], 40.00th=[ 50], 50.00th=[ 52], 60.00th=[ 54], | 70.00th=[ 57], 80.00th=[ 63], 90.00th=[ 78], 95.00th=[ 102], | 99.00th=[ 188], 99.50th=[ 318], 99.90th=[ 1020], 99.95th=[ 1434], | 99.99th=[ 2802] bw ( KiB/s): min=43065, max=53648, per=33.32%, avg=48300.73, stdev=2259.26, samples=358 iops : min=10766, max=13412, avg=12075.16, stdev=564.82, samples=358 lat (usec) : 50=42.34%, 100=52.29%, 250=4.74%, 500=0.25%, 750=0.18% lat (usec) : 1000=0.09% lat (msec) : 2=0.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=3560, avg=18.90, stdev= 8.41 sync percentiles (usec): | 1.00th=[ 10], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 14], | 30.00th=[ 15], 40.00th=[ 16], 50.00th=[ 17], 60.00th=[ 18], | 70.00th=[ 21], 80.00th=[ 25], 90.00th=[ 30], 95.00th=[ 34], | 99.00th=[ 44], 99.50th=[ 50], 99.90th=[ 70], 99.95th=[ 80], | 99.99th=[ 122] cpu : usr=3.96%, sys=18.58%, ctx=4383071, majf=0, minf=94 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2174555,0,2174552 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=142MiB/s (148MB/s), 142MiB/s-142MiB/s (148MB/s-148MB/s), io=8494MiB (8907MB), run=60001-60001msec Disk stats (read/write): sdg: ios=41/4337543, merge=0/1633, ticks=6/142118, in_queue=111072, util=94.38% s2p6-mx500-4k-sync-write-4: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 4 processes s2p6-mx500-4k-sync-write-4: (groupid=0, jobs=4): err= 0: pid=763176: Sat Apr 15 14:14:30 2023 write: IOPS=45.4k, BW=177MiB/s (186MB/s)(10.4GiB/60001msec); 0 zone resets clat (usec): min=36, max=26268, avg=62.93, stdev=98.32 lat (usec): min=36, max=26268, avg=63.15, stdev=98.33 clat percentiles (usec): | 1.00th=[ 43], 5.00th=[ 45], 10.00th=[ 46], 20.00th=[ 48], | 30.00th=[ 50], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 57], | 70.00th=[ 60], 80.00th=[ 64], 90.00th=[ 74], 95.00th=[ 95], | 99.00th=[ 182], 99.50th=[ 367], 99.90th=[ 1037], 99.95th=[ 1418], | 99.99th=[ 2802] bw ( KiB/s): min=39568, max=50136, per=24.99%, avg=45380.59, stdev=1762.89, samples=476 iops : min= 9890, max=12534, avg=11345.12, stdev=440.75, samples=476 lat (usec) : 50=29.74%, 100=66.07%, 250=3.50%, 500=0.29%, 750=0.18% lat (usec) : 1000=0.11% lat (msec) : 2=0.09%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=4497, avg=22.90, stdev=12.47 sync percentiles (usec): | 1.00th=[ 12], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 23], | 70.00th=[ 26], 80.00th=[ 31], 90.00th=[ 36], 95.00th=[ 41], | 99.00th=[ 55], 99.50th=[ 63], 99.90th=[ 104], 99.95th=[ 159], | 99.99th=[ 273] cpu : usr=3.91%, sys=18.77%, ctx=5528301, majf=0, minf=103 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2724435,0,2724432 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=177MiB/s (186MB/s), 177MiB/s-177MiB/s (186MB/s-186MB/s), io=10.4GiB (11.2GB), run=60001-60001msec Disk stats (read/write): sdg: ios=51/5436535, merge=0/297, ticks=7/185410, in_queue=136280, util=98.20% s2p6-mx500-4k-sync-write-5: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 5 processes s2p6-mx500-4k-sync-write-5: (groupid=0, jobs=5): err= 0: pid=765717: Sat Apr 15 14:15:30 2023 write: IOPS=52.6k, BW=205MiB/s (215MB/s)(12.0GiB/60001msec); 0 zone resets clat (usec): min=37, max=21018, avg=65.94, stdev=93.60 lat (usec): min=37, max=21019, avg=66.16, stdev=93.61 clat percentiles (usec): | 1.00th=[ 45], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 51], | 30.00th=[ 53], 40.00th=[ 56], 50.00th=[ 58], 60.00th=[ 61], | 70.00th=[ 64], 80.00th=[ 69], 90.00th=[ 79], 95.00th=[ 100], | 99.00th=[ 169], 99.50th=[ 277], 99.90th=[ 930], 99.95th=[ 1254], | 99.99th=[ 2474] bw ( KiB/s): min=29152, max=46112, per=19.99%, avg=42049.21, stdev=2111.68, samples=595 iops : min= 7288, max=11528, avg=10512.28, stdev=527.92, samples=595 lat (usec) : 50=16.27%, 100=78.72%, 250=4.44%, 500=0.24%, 750=0.16% lat (usec) : 1000=0.08% lat (msec) : 2=0.07%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=2426, avg=26.85, stdev=13.57 sync percentiles (usec): | 1.00th=[ 12], 5.00th=[ 15], 10.00th=[ 16], 20.00th=[ 17], | 30.00th=[ 19], 40.00th=[ 22], 50.00th=[ 25], 60.00th=[ 28], | 70.00th=[ 32], 80.00th=[ 36], 90.00th=[ 42], 95.00th=[ 47], | 99.00th=[ 62], 99.50th=[ 72], 99.90th=[ 116], 99.95th=[ 163], | 99.99th=[ 297] cpu : usr=3.92%, sys=17.76%, ctx=6511264, majf=0, minf=172 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3154668,0,3154663 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=205MiB/s (215MB/s), 205MiB/s-205MiB/s (215MB/s-215MB/s), io=12.0GiB (12.9GB), run=60001-60001msec Disk stats (read/write): sdg: ios=65/6294812, merge=0/597, ticks=8/227929, in_queue=160688, util=99.44% s2p6-mx500-4k-sync-write-6: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 6 processes s2p6-mx500-4k-sync-write-6: (groupid=0, jobs=6): err= 0: pid=768703: Sat Apr 15 14:16:31 2023 write: IOPS=56.1k, BW=219MiB/s (230MB/s)(12.8GiB/60001msec); 0 zone resets clat (usec): min=39, max=39239, avg=70.65, stdev=102.82 lat (usec): min=39, max=39240, avg=70.87, stdev=102.82 clat percentiles (usec): | 1.00th=[ 46], 5.00th=[ 49], 10.00th=[ 51], 20.00th=[ 55], | 30.00th=[ 57], 40.00th=[ 60], 50.00th=[ 63], 60.00th=[ 66], | 70.00th=[ 70], 80.00th=[ 75], 90.00th=[ 86], 95.00th=[ 112], | 99.00th=[ 178], 99.50th=[ 293], 99.90th=[ 906], 99.95th=[ 1254], | 99.99th=[ 2343] bw ( KiB/s): min=27240, max=40264, per=16.69%, avg=37419.60, stdev=1826.10, samples=714 iops : min= 6810, max=10066, avg=9354.88, stdev=456.52, samples=714 lat (usec) : 50=7.77%, 100=85.93%, 250=5.70%, 500=0.25%, 750=0.18% lat (usec) : 1000=0.08% lat (msec) : 2=0.06%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=6, max=3382, avg=33.96, stdev=16.55 sync percentiles (usec): | 1.00th=[ 13], 5.00th=[ 16], 10.00th=[ 18], 20.00th=[ 22], | 30.00th=[ 26], 40.00th=[ 30], 50.00th=[ 33], 60.00th=[ 37], | 70.00th=[ 40], 80.00th=[ 45], 90.00th=[ 51], 95.00th=[ 58], | 99.00th=[ 76], 99.50th=[ 87], 99.90th=[ 149], 99.95th=[ 186], | 99.99th=[ 330] cpu : usr=3.58%, sys=17.28%, ctx=7151457, majf=0, minf=189 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3363697,0,3363693 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=219MiB/s (230MB/s), 219MiB/s-219MiB/s (230MB/s-230MB/s), io=12.8GiB (13.8GB), run=60001-60001msec Disk stats (read/write): sdg: ios=51/6714992, merge=0/473, ticks=6/271677, in_queue=181200, util=99.65% s2p6-mx500-4k-sync-write-7: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 7 processes s2p6-mx500-4k-sync-write-7: (groupid=0, jobs=7): err= 0: pid=771193: Sat Apr 15 14:17:32 2023 write: IOPS=58.2k, BW=227MiB/s (239MB/s)(13.3GiB/60001msec); 0 zone resets clat (usec): min=38, max=19439, avg=77.49, stdev=98.97 lat (usec): min=38, max=19439, avg=77.74, stdev=98.98 clat percentiles (usec): | 1.00th=[ 47], 5.00th=[ 50], 10.00th=[ 53], 20.00th=[ 57], | 30.00th=[ 60], 40.00th=[ 63], 50.00th=[ 67], 60.00th=[ 70], | 70.00th=[ 74], 80.00th=[ 81], 90.00th=[ 95], 95.00th=[ 129], | 99.00th=[ 231], 99.50th=[ 553], 99.90th=[ 1188], 99.95th=[ 1549], | 99.99th=[ 2933] bw ( KiB/s): min=24184, max=36032, per=14.28%, avg=33264.71, stdev=2732.02, samples=833 iops : min= 6046, max= 9008, avg=8316.16, stdev=683.01, samples=833 lat (usec) : 50=4.88%, 100=86.60%, 250=7.60%, 500=0.37%, 750=0.27% lat (usec) : 1000=0.15% lat (msec) : 2=0.10%, 4=0.03%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=3176, avg=40.43, stdev=18.03 sync percentiles (usec): | 1.00th=[ 15], 5.00th=[ 19], 10.00th=[ 22], 20.00th=[ 29], | 30.00th=[ 33], 40.00th=[ 36], 50.00th=[ 39], 60.00th=[ 43], | 70.00th=[ 46], 80.00th=[ 51], 90.00th=[ 59], 95.00th=[ 68], | 99.00th=[ 90], 99.50th=[ 101], 99.90th=[ 159], 99.95th=[ 194], | 99.99th=[ 351] cpu : usr=3.33%, sys=17.05%, ctx=7552480, majf=0, minf=207 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3493797,0,3493793 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=227MiB/s (239MB/s), 227MiB/s-227MiB/s (239MB/s-239MB/s), io=13.3GiB (14.3GB), run=60001-60001msec Disk stats (read/write): sdg: ios=56/6969848, merge=0/1451, ticks=10/319279, in_queue=206368, util=99.84% s2p6-mx500-4k-sync-write-8: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 8 processes s2p6-mx500-4k-sync-write-8: (groupid=0, jobs=8): err= 0: pid=773720: Sat Apr 15 14:18:33 2023 write: IOPS=63.3k, BW=247MiB/s (259MB/s)(14.5GiB/60001msec); 0 zone resets clat (usec): min=39, max=13832, avg=78.81, stdev=80.35 lat (usec): min=40, max=13832, avg=79.04, stdev=80.36 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 52], 10.00th=[ 56], 20.00th=[ 60], | 30.00th=[ 63], 40.00th=[ 67], 50.00th=[ 71], 60.00th=[ 74], | 70.00th=[ 79], 80.00th=[ 85], 90.00th=[ 98], 95.00th=[ 124], | 99.00th=[ 196], 99.50th=[ 269], 99.90th=[ 971], 99.95th=[ 1385], | 99.99th=[ 2671] bw ( KiB/s): min=26856, max=34336, per=12.50%, avg=31665.06, stdev=1007.81, samples=953 iops : min= 6714, max= 8584, avg=7916.24, stdev=251.95, samples=953 lat (usec) : 50=2.90%, 100=87.93%, 250=8.64%, 500=0.21%, 750=0.16% lat (usec) : 1000=0.08% lat (msec) : 2=0.07%, 4=0.02%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=5750, avg=45.13, stdev=22.08 sync percentiles (usec): | 1.00th=[ 16], 5.00th=[ 21], 10.00th=[ 26], 20.00th=[ 33], | 30.00th=[ 37], 40.00th=[ 40], 50.00th=[ 43], 60.00th=[ 47], | 70.00th=[ 51], 80.00th=[ 57], 90.00th=[ 65], 95.00th=[ 75], | 99.00th=[ 98], 99.50th=[ 112], 99.90th=[ 184], 99.95th=[ 258], | 99.99th=[ 660] cpu : usr=3.34%, sys=16.85%, ctx=8349763, majf=0, minf=277 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,3798803,0,3798798 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=14.5GiB (15.6GB), run=60001-60001msec Disk stats (read/write): sdg: ios=54/7580643, merge=0/1271, ticks=7/367507, in_queue=226584, util=99.72% s2p6-mx500-4k-sync-write-9: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 9 processes s2p6-mx500-4k-sync-write-9: (groupid=0, jobs=9): err= 0: pid=776827: Sat Apr 15 14:19:34 2023 write: IOPS=66.0k, BW=262MiB/s (274MB/s)(15.3GiB/60001msec); 0 zone resets clat (usec): min=40, max=20753, avg=83.36, stdev=85.40 lat (usec): min=40, max=20753, avg=83.59, stdev=85.41 clat percentiles (usec): | 1.00th=[ 49], 5.00th=[ 56], 10.00th=[ 59], 20.00th=[ 64], | 30.00th=[ 68], 40.00th=[ 71], 50.00th=[ 75], 60.00th=[ 79], | 70.00th=[ 84], 80.00th=[ 91], 90.00th=[ 105], 95.00th=[ 135], | 99.00th=[ 206], 99.50th=[ 273], 99.90th=[ 848], 99.95th=[ 1188], | 99.99th=[ 2474] bw ( KiB/s): min=24048, max=32560, per=11.11%, avg=29756.01, stdev=1050.08, samples=1071 iops : min= 6012, max= 8140, avg=7438.97, stdev=262.51, samples=1071 lat (usec) : 50=1.41%, 100=86.09%, 250=11.93%, 500=0.28%, 750=0.14% lat (usec) : 1000=0.08% lat (msec) : 2=0.05%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=4577, avg=48.82, stdev=21.54 sync percentiles (usec): | 1.00th=[ 17], 5.00th=[ 24], 10.00th=[ 31], 20.00th=[ 36], | 30.00th=[ 40], 40.00th=[ 43], 50.00th=[ 46], 60.00th=[ 50], | 70.00th=[ 55], 80.00th=[ 60], 90.00th=[ 71], 95.00th=[ 82], | 99.00th=[ 110], 99.50th=[ 125], 99.90th=[ 194], 99.95th=[ 249], | 99.99th=[ 570] cpu : usr=2.83%, sys=17.05%, ctx=8877027, majf=0, minf=276 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4017312,0,4017308 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=262MiB/s (274MB/s), 262MiB/s-262MiB/s (274MB/s-274MB/s), io=15.3GiB (16.5GB), run=60001-60001msec Disk stats (read/write): sdg: ios=56/8016295, merge=0/1653, ticks=7/418285, in_queue=253988, util=99.85% s2p6-mx500-4k-sync-write-10: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 10 processes s2p6-mx500-4k-sync-write-10: (groupid=0, jobs=10): err= 0: pid=779289: Sat Apr 15 14:20:34 2023 write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60001msec); 0 zone resets clat (usec): min=39, max=28230, avg=92.44, stdev=96.85 lat (usec): min=39, max=28231, avg=92.67, stdev=96.86 clat percentiles (usec): | 1.00th=[ 52], 5.00th=[ 59], 10.00th=[ 63], 20.00th=[ 69], | 30.00th=[ 73], 40.00th=[ 77], 50.00th=[ 81], 60.00th=[ 85], | 70.00th=[ 91], 80.00th=[ 99], 90.00th=[ 118], 95.00th=[ 155], | 99.00th=[ 281], 99.50th=[ 469], 99.90th=[ 1029], 99.95th=[ 1303], | 99.99th=[ 2147] bw ( KiB/s): min=20696, max=30872, per=10.00%, avg=27290.11, stdev=1274.40, samples=1191 iops : min= 5174, max= 7718, avg=6822.50, stdev=318.61, samples=1191 lat (usec) : 50=0.56%, 100=80.25%, 250=17.95%, 500=0.75%, 750=0.19% lat (usec) : 1000=0.19% lat (msec) : 2=0.10%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=3497, avg=51.81, stdev=24.35 sync percentiles (usec): | 1.00th=[ 18], 5.00th=[ 27], 10.00th=[ 33], 20.00th=[ 38], | 30.00th=[ 41], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 52], | 70.00th=[ 57], 80.00th=[ 64], 90.00th=[ 77], 95.00th=[ 89], | 99.00th=[ 119], 99.50th=[ 133], 99.90th=[ 192], 99.95th=[ 247], | 99.99th=[ 717] cpu : usr=2.83%, sys=16.51%, ctx=9121717, majf=0, minf=301 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4092647,0,4092637 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.8GB), run=60001-60001msec Disk stats (read/write): sdg: ios=57/8165473, merge=0/1796, ticks=6/471633, in_queue=291644, util=99.60% s2p6-mx500-4k-sync-write-11: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 11 processes s2p6-mx500-4k-sync-write-11: (groupid=0, jobs=11): err= 0: pid=781864: Sat Apr 15 14:21:35 2023 write: IOPS=68.4k, BW=267MiB/s (280MB/s)(15.7GiB/60002msec); 0 zone resets clat (usec): min=38, max=5072, avg=102.48, stdev=81.10 lat (usec): min=38, max=5073, avg=102.71, stdev=81.11 clat percentiles (usec): | 1.00th=[ 55], 5.00th=[ 63], 10.00th=[ 68], 20.00th=[ 74], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 92], | 70.00th=[ 99], 80.00th=[ 111], 90.00th=[ 137], 95.00th=[ 186], | 99.00th=[ 367], 99.50th=[ 742], 99.90th=[ 1139], 99.95th=[ 1336], | 99.99th=[ 1778] bw ( KiB/s): min=19256, max=28351, per=9.09%, avg=24872.07, stdev=1362.87, samples=1310 iops : min= 4814, max= 7087, avg=6217.99, stdev=340.72, samples=1310 lat (usec) : 50=0.33%, 100=70.95%, 250=26.72%, 500=1.24%, 750=0.29% lat (usec) : 1000=0.32% lat (msec) : 2=0.14%, 4=0.01%, 10=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=6223, avg=55.83, stdev=30.27 sync percentiles (usec): | 1.00th=[ 18], 5.00th=[ 29], 10.00th=[ 35], 20.00th=[ 40], | 30.00th=[ 44], 40.00th=[ 47], 50.00th=[ 51], 60.00th=[ 56], | 70.00th=[ 61], 80.00th=[ 70], 90.00th=[ 85], 95.00th=[ 98], | 99.00th=[ 135], 99.50th=[ 153], 99.90th=[ 249], 99.95th=[ 351], | 99.99th=[ 955] cpu : usr=2.61%, sys=16.48%, ctx=9194667, majf=0, minf=337 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4103460,0,4103453 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=15.7GiB (16.8GB), run=60002-60002msec Disk stats (read/write): sdg: ios=56/8185172, merge=0/2261, ticks=7/523597, in_queue=326748, util=99.37% s2p6-mx500-4k-sync-write-12: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 12 processes s2p6-mx500-4k-sync-write-12: (groupid=0, jobs=12): err= 0: pid=784754: Sat Apr 15 14:22:36 2023 write: IOPS=72.8k, BW=284MiB/s (298MB/s)(16.7GiB/60001msec); 0 zone resets clat (usec): min=39, max=9327, avg=104.15, stdev=65.97 lat (usec): min=40, max=9327, avg=104.40, stdev=65.99 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 68], 10.00th=[ 73], 20.00th=[ 80], | 30.00th=[ 85], 40.00th=[ 90], 50.00th=[ 94], 60.00th=[ 99], | 70.00th=[ 105], 80.00th=[ 115], 90.00th=[ 135], 95.00th=[ 167], | 99.00th=[ 260], 99.50th=[ 363], 99.90th=[ 971], 99.95th=[ 1237], | 99.99th=[ 1729] bw ( KiB/s): min=21600, max=26650, per=8.33%, avg=24255.13, stdev=923.84, samples=1429 iops : min= 5400, max= 6662, avg=6063.76, stdev=230.95, samples=1429 lat (usec) : 50=0.16%, 100=61.43%, 250=37.27%, 500=0.74%, 750=0.14% lat (usec) : 1000=0.17% lat (msec) : 2=0.09%, 4=0.01%, 10=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=8258, avg=58.21, stdev=30.53 sync percentiles (usec): | 1.00th=[ 20], 5.00th=[ 32], 10.00th=[ 36], 20.00th=[ 41], | 30.00th=[ 45], 40.00th=[ 48], 50.00th=[ 52], 60.00th=[ 58], | 70.00th=[ 64], 80.00th=[ 73], 90.00th=[ 89], 95.00th=[ 105], | 99.00th=[ 143], 99.50th=[ 163], 99.90th=[ 245], 99.95th=[ 338], | 99.99th=[ 906] cpu : usr=2.53%, sys=16.92%, ctx=9886578, majf=0, minf=397 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4367387,0,4367378 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=284MiB/s (298MB/s), 284MiB/s-284MiB/s (298MB/s-298MB/s), io=16.7GiB (17.9GB), run=60001-60001msec Disk stats (read/write): sdg: ios=67/8711483, merge=0/2742, ticks=9/573982, in_queue=352060, util=99.50% s2p6-mx500-4k-sync-write-13: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 13 processes s2p6-mx500-4k-sync-write-13: (groupid=0, jobs=13): err= 0: pid=787422: Sat Apr 15 14:23:38 2023 write: IOPS=75.4k, BW=295MiB/s (309MB/s)(17.3GiB/60001msec); 0 zone resets clat (usec): min=38, max=18536, avg=111.28, stdev=82.74 lat (usec): min=38, max=18536, avg=111.51, stdev=82.75 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 74], 10.00th=[ 80], 20.00th=[ 88], | 30.00th=[ 93], 40.00th=[ 98], 50.00th=[ 103], 60.00th=[ 109], | 70.00th=[ 115], 80.00th=[ 123], 90.00th=[ 141], 95.00th=[ 167], | 99.00th=[ 249], 99.50th=[ 289], 99.90th=[ 857], 99.95th=[ 1172], | 99.99th=[ 2507] bw ( KiB/s): min=20488, max=24976, per=7.69%, avg=23204.69, stdev=633.31, samples=1547 iops : min= 5122, max= 6242, avg=5801.16, stdev=158.34, samples=1547 lat (usec) : 50=0.07%, 100=43.89%, 250=55.07%, 500=0.69%, 750=0.09% lat (usec) : 1000=0.11% lat (msec) : 2=0.06%, 4=0.01%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=7313, avg=58.89, stdev=30.88 sync percentiles (usec): | 1.00th=[ 20], 5.00th=[ 31], 10.00th=[ 35], 20.00th=[ 40], | 30.00th=[ 44], 40.00th=[ 48], 50.00th=[ 52], 60.00th=[ 58], | 70.00th=[ 65], 80.00th=[ 75], 90.00th=[ 92], 95.00th=[ 111], | 99.00th=[ 151], 99.50th=[ 172], 99.90th=[ 235], 99.95th=[ 302], | 99.99th=[ 807] cpu : usr=2.22%, sys=16.80%, ctx=10274698, majf=0, minf=468 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4523643,0,4523634 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=295MiB/s (309MB/s), 295MiB/s-295MiB/s (309MB/s-309MB/s), io=17.3GiB (18.5GB), run=60001-60001msec Disk stats (read/write): sdg: ios=51/9043694, merge=0/2325, ticks=6/629447, in_queue=397892, util=99.47% s2p6-mx500-4k-sync-write-14: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 14 processes s2p6-mx500-4k-sync-write-14: (groupid=0, jobs=14): err= 0: pid=790037: Sat Apr 15 14:24:39 2023 write: IOPS=73.2k, BW=286MiB/s (300MB/s)(16.8GiB/60001msec); 0 zone resets clat (usec): min=39, max=26990, avg=121.54, stdev=101.06 lat (usec): min=39, max=26991, avg=121.74, stdev=101.06 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 78], 10.00th=[ 85], 20.00th=[ 93], | 30.00th=[ 100], 40.00th=[ 105], 50.00th=[ 112], 60.00th=[ 118], | 70.00th=[ 125], 80.00th=[ 137], 90.00th=[ 159], 95.00th=[ 194], | 99.00th=[ 281], 99.50th=[ 355], 99.90th=[ 947], 99.95th=[ 1336], | 99.99th=[ 2573] bw ( KiB/s): min=16056, max=23368, per=7.14%, avg=20916.95, stdev=820.27, samples=1666 iops : min= 4014, max= 5842, avg=5229.22, stdev=205.07, samples=1666 lat (usec) : 50=0.07%, 100=30.70%, 250=67.51%, 500=1.36%, 750=0.12% lat (usec) : 1000=0.15% lat (msec) : 2=0.07%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=4883, avg=67.39, stdev=35.09 sync percentiles (usec): | 1.00th=[ 22], 5.00th=[ 34], 10.00th=[ 39], 20.00th=[ 44], | 30.00th=[ 48], 40.00th=[ 53], 50.00th=[ 59], 60.00th=[ 65], | 70.00th=[ 75], 80.00th=[ 88], 90.00th=[ 111], 95.00th=[ 133], | 99.00th=[ 180], 99.50th=[ 200], 99.90th=[ 255], 99.95th=[ 293], | 99.99th=[ 865] cpu : usr=1.97%, sys=18.14%, ctx=10137472, majf=0, minf=493 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4393649,0,4393640 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=286MiB/s (300MB/s), 286MiB/s-286MiB/s (300MB/s-300MB/s), io=16.8GiB (17.0GB), run=60001-60001msec Disk stats (read/write): sdg: ios=18/8763875, merge=0/3217, ticks=4/678677, in_queue=412480, util=98.77% s2p6-mx500-4k-sync-write-15: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 15 processes s2p6-mx500-4k-sync-write-15: (groupid=0, jobs=15): err= 0: pid=792627: Sat Apr 15 14:25:39 2023 write: IOPS=72.0k, BW=281MiB/s (295MB/s)(16.5GiB/60002msec); 0 zone resets clat (usec): min=37, max=55834, avg=133.83, stdev=145.33 lat (usec): min=37, max=55835, avg=134.07, stdev=145.33 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 82], 10.00th=[ 90], 20.00th=[ 100], | 30.00th=[ 109], 40.00th=[ 115], 50.00th=[ 121], 60.00th=[ 128], | 70.00th=[ 137], 80.00th=[ 149], 90.00th=[ 176], 95.00th=[ 212], | 99.00th=[ 330], 99.50th=[ 586], 99.90th=[ 1385], 99.95th=[ 1827], | 99.99th=[ 3032] bw ( KiB/s): min=13900, max=21352, per=6.66%, avg=19192.42, stdev=1093.06, samples=1786 iops : min= 3475, max= 5338, avg=4798.09, stdev=273.27, samples=1786 lat (usec) : 50=0.06%, 100=19.93%, 250=77.31%, 500=2.14%, 750=0.13% lat (usec) : 1000=0.24% lat (msec) : 2=0.14%, 4=0.04%, 10=0.01%, 20=0.01%, 100=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=7, max=3896, avg=72.09, stdev=41.44 sync percentiles (usec): | 1.00th=[ 23], 5.00th=[ 35], 10.00th=[ 39], 20.00th=[ 45], | 30.00th=[ 50], 40.00th=[ 56], 50.00th=[ 62], 60.00th=[ 70], | 70.00th=[ 80], 80.00th=[ 95], 90.00th=[ 121], 95.00th=[ 145], | 99.00th=[ 198], 99.50th=[ 221], 99.90th=[ 281], 99.95th=[ 330], | 99.99th=[ 979] cpu : usr=1.91%, sys=18.36%, ctx=10081528, majf=0, minf=544 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4320254,0,4320244 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=281MiB/s (295MB/s), 281MiB/s-281MiB/s (295MB/s-295MB/s), io=16.5GiB (17.7GB), run=60002-60002msec Disk stats (read/write): sdg: ios=51/8615350, merge=0/4226, ticks=6/733589, in_queue=448132, util=98.59% s2p6-mx500-4k-sync-write-16: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1 ... fio-3.12 Starting 16 processes s2p6-mx500-4k-sync-write-16: (groupid=0, jobs=16): err= 0: pid=795502: Sat Apr 15 14:26:40 2023 write: IOPS=72.4k, BW=283MiB/s (296MB/s)(16.6GiB/60001msec); 0 zone resets clat (usec): min=40, max=18042, avg=141.75, stdev=92.29 lat (usec): min=40, max=18042, avg=141.99, stdev=92.30 clat percentiles (usec): | 1.00th=[ 70], 5.00th=[ 87], 10.00th=[ 96], 20.00th=[ 109], | 30.00th=[ 117], 40.00th=[ 124], 50.00th=[ 131], 60.00th=[ 139], | 70.00th=[ 147], 80.00th=[ 161], 90.00th=[ 190], 95.00th=[ 227], | 99.00th=[ 314], 99.50th=[ 371], 99.90th=[ 1106], 99.95th=[ 1467], | 99.99th=[ 2671] bw ( KiB/s): min=15393, max=19976, per=6.25%, avg=18091.88, stdev=613.54, samples=1906 iops : min= 3848, max= 4994, avg=4522.95, stdev=153.38, samples=1906 lat (usec) : 50=0.05%, 100=12.63%, 250=84.06%, 500=2.88%, 750=0.10% lat (usec) : 1000=0.15% lat (msec) : 2=0.10%, 4=0.02%, 10=0.01%, 20=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=8, max=6261, avg=76.81, stdev=45.21 sync percentiles (usec): | 1.00th=[ 23], 5.00th=[ 35], 10.00th=[ 40], 20.00th=[ 46], | 30.00th=[ 51], 40.00th=[ 57], 50.00th=[ 64], 60.00th=[ 74], | 70.00th=[ 86], 80.00th=[ 104], 90.00th=[ 135], 95.00th=[ 161], | 99.00th=[ 221], 99.50th=[ 245], 99.90th=[ 318], 99.95th=[ 375], | 99.99th=[ 881] cpu : usr=1.82%, sys=19.04%, ctx=10230565, majf=0, minf=532 IO depths : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4343279,0,4343267 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=283MiB/s (296MB/s), 283MiB/s-283MiB/s (296MB/s-296MB/s), io=16.6GiB (17.8GB), run=60001-60001msec Disk stats (read/write): sdg: ios=17/8662198, merge=0/4196, ticks=3/780266, in_queue=467296, util=97.39%