s6p6-mx500-4k-sync-write-1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
fio-3.12
Starting 1 process

s6p6-mx500-4k-sync-write-1: (groupid=0, jobs=1): err= 0: pid=1952241: Sat Apr 15 13:49:59 2023
  write: IOPS=18.8k, BW=73.5MiB/s (77.1MB/s)(4410MiB/60001msec); 0 zone resets
    clat (usec): min=33, max=4485, avg=43.33, stdev=37.63
     lat (usec): min=33, max=4485, avg=43.51, stdev=37.63
    clat percentiles (usec):
     |  1.00th=[   37],  5.00th=[   37], 10.00th=[   38], 20.00th=[   38],
     | 30.00th=[   38], 40.00th=[   38], 50.00th=[   39], 60.00th=[   39],
     | 70.00th=[   39], 80.00th=[   41], 90.00th=[   53], 95.00th=[   65],
     | 99.00th=[   92], 99.50th=[  130], 99.90th=[  668], 99.95th=[  791],
     | 99.99th=[ 1172]
   bw (  KiB/s): min=65304, max=77600, per=99.98%, avg=75252.20, stdev=1806.87, samples=119
   iops        : min=16326, max=19400, avg=18813.06, stdev=451.76, samples=119
  lat (usec)   : 50=88.81%, 100=10.40%, 250=0.51%, 500=0.06%, 750=0.16%
  lat (usec)   : 1000=0.03%
  lat (msec)   : 2=0.02%, 4=0.01%, 10=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=891, avg= 7.61, stdev= 2.28
    sync percentiles (nsec):
     |  1.00th=[ 7008],  5.00th=[ 7136], 10.00th=[ 7136], 20.00th=[ 7200],
     | 30.00th=[ 7264], 40.00th=[ 7328], 50.00th=[ 7328], 60.00th=[ 7392],
     | 70.00th=[ 7392], 80.00th=[ 7520], 90.00th=[ 7712], 95.00th=[ 8512],
     | 99.00th=[12480], 99.50th=[18816], 99.90th=[33536], 99.95th=[38656],
     | 99.99th=[55040]
  cpu          : usr=7.72%, sys=15.52%, ctx=2258215, majf=0, minf=27
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1129079,0,1129078 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=73.5MiB/s (77.1MB/s), 73.5MiB/s-73.5MiB/s (77.1MB/s-77.1MB/s), io=4410MiB (4625MB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=0/2253300, merge=0/0, ticks=0/48447, in_queue=41976, util=70.01%
s6p6-mx500-4k-sync-write-2: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 2 processes

s6p6-mx500-4k-sync-write-2: (groupid=0, jobs=2): err= 0: pid=1953821: Sat Apr 15 13:51:00 2023
  write: IOPS=27.6k, BW=108MiB/s (113MB/s)(6474MiB/60001msec); 0 zone resets
    clat (usec): min=32, max=13339, avg=54.55, stdev=44.62
     lat (usec): min=33, max=13340, avg=54.78, stdev=44.63
    clat percentiles (usec):
     |  1.00th=[   40],  5.00th=[   42], 10.00th=[   43], 20.00th=[   45],
     | 30.00th=[   49], 40.00th=[   50], 50.00th=[   51], 60.00th=[   51],
     | 70.00th=[   52], 80.00th=[   54], 90.00th=[   62], 95.00th=[   81],
     | 99.00th=[  113], 99.50th=[  153], 99.90th=[  693], 99.95th=[  898],
     | 99.99th=[ 1319]
   bw (  KiB/s): min=48784, max=61512, per=49.98%, avg=55222.97, stdev=1874.90, samples=238
   iops        : min=12196, max=15378, avg=13805.75, stdev=468.74, samples=238
  lat (usec)   : 50=44.78%, 100=53.14%, 250=1.76%, 500=0.07%, 750=0.17%
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.03%, 4=0.01%, 10=0.01%, 20=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=380, avg=15.65, stdev= 4.16
    sync percentiles (nsec):
     |  1.00th=[ 9408],  5.00th=[10688], 10.00th=[12864], 20.00th=[13888],
     | 30.00th=[14272], 40.00th=[14528], 50.00th=[14784], 60.00th=[15040],
     | 70.00th=[15296], 80.00th=[16192], 90.00th=[20352], 95.00th=[24704],
     | 99.00th=[30592], 99.50th=[35072], 99.90th=[48384], 99.95th=[54528],
     | 99.99th=[70144]
  cpu          : usr=4.65%, sys=20.18%, ctx=3317258, majf=0, minf=65
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1657249,0,1657247 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=108MiB/s (113MB/s), 108MiB/s-108MiB/s (113MB/s-113MB/s), io=6474MiB (6788MB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=102/3306250, merge=0/962, ticks=12/95817, in_queue=75976, util=80.78%
s6p6-mx500-4k-sync-write-3: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 3 processes

s6p6-mx500-4k-sync-write-3: (groupid=0, jobs=3): err= 0: pid=1955336: Sat Apr 15 13:52:01 2023
  write: IOPS=29.0k, BW=117MiB/s (123MB/s)(7020MiB/60001msec); 0 zone resets
    clat (usec): min=34, max=1639.3k, avg=79.75, stdev=2698.31
     lat (usec): min=34, max=1639.3k, avg=79.97, stdev=2698.31
    clat percentiles (usec):
     |  1.00th=[   41],  5.00th=[   42], 10.00th=[   43], 20.00th=[   45],
     | 30.00th=[   47], 40.00th=[   49], 50.00th=[   51], 60.00th=[   52],
     | 70.00th=[   55], 80.00th=[   59], 90.00th=[   71], 95.00th=[   98],
     | 99.00th=[  130], 99.50th=[  200], 99.90th=[ 1020], 99.95th=[ 2114],
     | 99.99th=[78119]
   bw (  KiB/s): min= 5216, max=56912, per=33.91%, avg=40628.95, stdev=12972.90, samples=351
   iops        : min= 1304, max=14230, avg=10157.22, stdev=3243.22, samples=351
  lat (usec)   : 50=48.07%, 100=47.35%, 250=4.17%, 500=0.10%, 750=0.17%
  lat (usec)   : 1000=0.05%
  lat (msec)   : 2=0.05%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=1977, avg=18.21, stdev= 7.21
    sync percentiles (nsec):
     |  1.00th=[ 9664],  5.00th=[10688], 10.00th=[11584], 20.00th=[12992],
     | 30.00th=[14272], 40.00th=[15168], 50.00th=[15808], 60.00th=[17024],
     | 70.00th=[19584], 80.00th=[23424], 90.00th=[28288], 95.00th=[31616],
     | 99.00th=[39680], 99.50th=[44800], 99.90th=[60160], 99.95th=[67072],
     | 99.99th=[88576]
  cpu          : usr=3.07%, sys=15.25%, ctx=3618409, majf=0, minf=103
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1797222,0,1797220 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=117MiB/s (123MB/s), 117MiB/s-117MiB/s (123MB/s-123MB/s), io=7020MiB (7361MB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/3580812, merge=0/3012, ticks=6/149555, in_queue=124512, util=95.34%
s6p6-mx500-4k-sync-write-4: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 4 processes

s6p6-mx500-4k-sync-write-4: (groupid=0, jobs=4): err= 0: pid=1956701: Sat Apr 15 13:53:01 2023
  write: IOPS=30.3k, BW=118MiB/s (124MB/s)(7099MiB/60001msec); 0 zone resets
    clat (usec): min=37, max=2204.3k, avg=109.40, stdev=4814.41
     lat (usec): min=37, max=2204.3k, avg=109.60, stdev=4814.41
    clat percentiles (usec):
     |  1.00th=[    43],  5.00th=[    45], 10.00th=[    46], 20.00th=[    48],
     | 30.00th=[    50], 40.00th=[    51], 50.00th=[    53], 60.00th=[    55],
     | 70.00th=[    58], 80.00th=[    62], 90.00th=[    70], 95.00th=[    90],
     | 99.00th=[   123], 99.50th=[   165], 99.90th=[  1020], 99.95th=[  2212],
     | 99.99th=[152044]
   bw (  KiB/s): min= 2512, max=51576, per=26.26%, avg=31815.82, stdev=13461.03, samples=452
   iops        : min=  628, max=12894, avg=7953.94, stdev=3365.28, samples=452
  lat (usec)   : 50=33.82%, 100=62.81%, 250=3.06%, 500=0.05%, 750=0.14%
  lat (usec)   : 1000=0.03%
  lat (msec)   : 2=0.05%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=625, avg=20.52, stdev= 8.24
    sync percentiles (usec):
     |  1.00th=[   11],  5.00th=[   13], 10.00th=[   13], 20.00th=[   15],
     | 30.00th=[   16], 40.00th=[   17], 50.00th=[   18], 60.00th=[   20],
     | 70.00th=[   23], 80.00th=[   27], 90.00th=[   32], 95.00th=[   37],
     | 99.00th=[   48], 99.50th=[   53], 99.90th=[   70], 99.95th=[   78],
     | 99.99th=[  101]
  cpu          : usr=2.48%, sys=12.03%, ctx=3676676, majf=0, minf=130
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1817384,0,1817383 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=118MiB/s (124MB/s), 118MiB/s-118MiB/s (124MB/s-124MB/s), io=7099MiB (7444MB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/3621568, merge=0/428, ticks=6/206097, in_queue=177640, util=99.71%
s6p6-mx500-4k-sync-write-5: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 5 processes

s6p6-mx500-4k-sync-write-5: (groupid=0, jobs=5): err= 0: pid=1958011: Sat Apr 15 13:54:02 2023
  write: IOPS=37.9k, BW=148MiB/s (155MB/s)(8886MiB/60062msec); 0 zone resets
    clat (usec): min=38, max=474870, avg=103.29, stdev=2478.08
     lat (usec): min=38, max=474870, avg=103.52, stdev=2478.08
    clat percentiles (usec):
     |  1.00th=[    45],  5.00th=[    47], 10.00th=[    48], 20.00th=[    51],
     | 30.00th=[    53], 40.00th=[    56], 50.00th=[    58], 60.00th=[    60],
     | 70.00th=[    63], 80.00th=[    68], 90.00th=[    78], 95.00th=[    97],
     | 99.00th=[   145], 99.50th=[   200], 99.90th=[  1156], 99.95th=[  2573],
     | 99.99th=[141558]
   bw (  KiB/s): min= 9288, max=46232, per=20.02%, avg=30326.43, stdev=10358.54, samples=600
   iops        : min= 2322, max=11558, avg=7581.59, stdev=2589.64, samples=600
  lat (usec)   : 50=17.44%, 100=78.07%, 250=4.11%, 500=0.09%, 750=0.14%
  lat (usec)   : 1000=0.04%
  lat (msec)   : 2=0.05%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.02%, 500=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=1682, avg=26.34, stdev=11.29
    sync percentiles (usec):
     |  1.00th=[   12],  5.00th=[   15], 10.00th=[   16], 20.00th=[   17],
     | 30.00th=[   19], 40.00th=[   21], 50.00th=[   25], 60.00th=[   28],
     | 70.00th=[   31], 80.00th=[   36], 90.00th=[   41], 95.00th=[   46],
     | 99.00th=[   59], 99.50th=[   65], 99.90th=[   83], 99.95th=[   92],
     | 99.99th=[  149]
  cpu          : usr=2.52%, sys=12.95%, ctx=4695633, majf=0, minf=155
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2274833,0,2274828 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=148MiB/s (155MB/s), 148MiB/s-148MiB/s (155MB/s-155MB/s), io=8886MiB (9318MB), run=60062-60062msec

Disk stats (read/write):
  sdl: ios=0/4542874, merge=0/934, ticks=0/248499, in_queue=201164, util=99.59%
s6p6-mx500-4k-sync-write-6: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 6 processes

s6p6-mx500-4k-sync-write-6: (groupid=0, jobs=6): err= 0: pid=1959669: Sat Apr 15 13:55:03 2023
  write: IOPS=39.5k, BW=154MiB/s (162MB/s)(9255MiB/60001msec); 0 zone resets
    clat (usec): min=36, max=1446.0k, avg=117.08, stdev=3818.35
     lat (usec): min=36, max=1446.0k, avg=117.32, stdev=3818.36
    clat percentiles (usec):
     |  1.00th=[    46],  5.00th=[    49], 10.00th=[    51], 20.00th=[    55],
     | 30.00th=[    57], 40.00th=[    60], 50.00th=[    62], 60.00th=[    65],
     | 70.00th=[    69], 80.00th=[    74], 90.00th=[    84], 95.00th=[   108],
     | 99.00th=[   172], 99.50th=[   249], 99.90th=[  1450], 99.95th=[  2671],
     | 99.99th=[145753]
   bw (  KiB/s): min= 2568, max=40950, per=17.04%, avg=26914.22, stdev=9890.85, samples=702
   iops        : min=  642, max=10237, avg=6728.54, stdev=2472.72, samples=702
  lat (usec)   : 50=8.01%, 100=86.19%, 250=5.30%, 500=0.16%, 750=0.14%
  lat (usec)   : 1000=0.05%
  lat (msec)   : 2=0.08%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.02%, 500=0.01%, 750=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=1454, avg=32.56, stdev=13.75
    sync percentiles (usec):
     |  1.00th=[   13],  5.00th=[   16], 10.00th=[   18], 20.00th=[   21],
     | 30.00th=[   25], 40.00th=[   29], 50.00th=[   32], 60.00th=[   35],
     | 70.00th=[   39], 80.00th=[   43], 90.00th=[   49], 95.00th=[   56],
     | 99.00th=[   71], 99.50th=[   79], 99.90th=[   99], 99.95th=[  114],
     | 99.99th=[  237]
  cpu          : usr=2.41%, sys=12.18%, ctx=5017702, majf=0, minf=180
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,2369152,0,2369147 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=154MiB/s (162MB/s), 154MiB/s-154MiB/s (162MB/s-162MB/s), io=9255MiB (9704MB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/4735754, merge=0/1259, ticks=6/298590, in_queue=237172, util=99.96%
s6p6-mx500-4k-sync-write-7: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 7 processes

s6p6-mx500-4k-sync-write-7: (groupid=0, jobs=7): err= 0: pid=1961007: Sat Apr 15 13:56:06 2023
  write: IOPS=57.7k, BW=225MiB/s (236MB/s)(13.6GiB/61884msec); 0 zone resets
    clat (usec): min=38, max=2022.2k, avg=80.74, stdev=2983.52
     lat (usec): min=39, max=2022.2k, avg=80.95, stdev=2983.52
    clat percentiles (usec):
     |  1.00th=[   46],  5.00th=[   50], 10.00th=[   53], 20.00th=[   57],
     | 30.00th=[   60], 40.00th=[   62], 50.00th=[   65], 60.00th=[   69],
     | 70.00th=[   73], 80.00th=[   78], 90.00th=[   89], 95.00th=[  117],
     | 99.00th=[  188], 99.50th=[  269], 99.90th=[ 1090], 99.95th=[ 1582],
     | 99.99th=[ 3097]
   bw (  KiB/s): min= 3000, max=37800, per=14.73%, avg=33982.17, stdev=4130.31, samples=840
   iops        : min=  750, max= 9450, avg=8495.53, stdev=1032.58, samples=840
  lat (usec)   : 50=5.49%, 100=87.79%, 250=6.17%, 500=0.23%, 750=0.14%
  lat (usec)   : 1000=0.06%
  lat (msec)   : 2=0.08%, 4=0.03%, 10=0.01%, 50=0.01%, 100=0.01%
  lat (msec)   : 250=0.01%, 500=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=2270, avg=38.42, stdev=16.07
    sync percentiles (usec):
     |  1.00th=[   15],  5.00th=[   18], 10.00th=[   21], 20.00th=[   27],
     | 30.00th=[   32], 40.00th=[   35], 50.00th=[   38], 60.00th=[   41],
     | 70.00th=[   45], 80.00th=[   49], 90.00th=[   56], 95.00th=[   62],
     | 99.00th=[   79], 99.50th=[   87], 99.90th=[  115], 99.95th=[  141],
     | 99.99th=[  392]
  cpu          : usr=2.86%, sys=16.03%, ctx=7731050, majf=0, minf=213
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3568417,0,3568410 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=225MiB/s (236MB/s), 225MiB/s-225MiB/s (236MB/s-236MB/s), io=13.6GiB (14.6GB), run=61884-61884msec

Disk stats (read/write):
  sdl: ios=51/7135242, merge=0/1317, ticks=6/323162, in_queue=227212, util=100.00%
s6p6-mx500-4k-sync-write-8: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 8 processes

s6p6-mx500-4k-sync-write-8: (groupid=0, jobs=8): err= 0: pid=1962424: Sat Apr 15 13:57:06 2023
  write: IOPS=63.4k, BW=248MiB/s (260MB/s)(14.5GiB/60001msec); 0 zone resets
    clat (usec): min=40, max=180431, avg=80.96, stdev=404.24
     lat (usec): min=40, max=180432, avg=81.17, stdev=404.24
    clat percentiles (usec):
     |  1.00th=[   48],  5.00th=[   53], 10.00th=[   56], 20.00th=[   60],
     | 30.00th=[   63], 40.00th=[   67], 50.00th=[   70], 60.00th=[   74],
     | 70.00th=[   78], 80.00th=[   84], 90.00th=[   96], 95.00th=[  120],
     | 99.00th=[  202], 99.50th=[  293], 99.90th=[ 1287], 99.95th=[ 1975],
     | 99.99th=[ 7373]
   bw (  KiB/s): min=23544, max=35408, per=12.51%, avg=31720.13, stdev=1728.66, samples=952
   iops        : min= 5886, max= 8852, avg=7930.02, stdev=432.16, samples=952
  lat (usec)   : 50=2.81%, 100=88.80%, 250=7.76%, 500=0.28%, 750=0.13%
  lat (usec)   : 1000=0.07%
  lat (msec)   : 2=0.09%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 250=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=2354, avg=42.91, stdev=18.16
    sync percentiles (usec):
     |  1.00th=[   16],  5.00th=[   20], 10.00th=[   25], 20.00th=[   32],
     | 30.00th=[   36], 40.00th=[   39], 50.00th=[   42], 60.00th=[   45],
     | 70.00th=[   49], 80.00th=[   54], 90.00th=[   62], 95.00th=[   70],
     | 99.00th=[   90], 99.50th=[   99], 99.90th=[  133], 99.95th=[  172],
     | 99.99th=[  482]
  cpu          : usr=3.00%, sys=16.63%, ctx=8366051, majf=0, minf=288
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3802949,0,3802942 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=248MiB/s (260MB/s), 248MiB/s-248MiB/s (260MB/s-260MB/s), io=14.5GiB (15.6GB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/7586310, merge=0/1191, ticks=10/370280, in_queue=237576, util=100.00%
s6p6-mx500-4k-sync-write-9: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 9 processes

s6p6-mx500-4k-sync-write-9: (groupid=0, jobs=9): err= 0: pid=1964115: Sat Apr 15 13:58:07 2023
  write: IOPS=62.7k, BW=245MiB/s (257MB/s)(14.3GiB/60001msec); 0 zone resets
    clat (usec): min=39, max=1315.6k, avg=94.64, stdev=2374.38
     lat (usec): min=40, max=1315.6k, avg=94.87, stdev=2374.38
    clat percentiles (usec):
     |  1.00th=[   50],  5.00th=[   56], 10.00th=[   59], 20.00th=[   63],
     | 30.00th=[   67], 40.00th=[   71], 50.00th=[   74], 60.00th=[   78],
     | 70.00th=[   82], 80.00th=[   89], 90.00th=[  102], 95.00th=[  126],
     | 99.00th=[  206], 99.50th=[  310], 99.90th=[ 1450], 99.95th=[ 2376],
     | 99.99th=[30278]
   bw (  KiB/s): min=  656, max=32646, per=11.19%, avg=28074.25, stdev=4907.62, samples=1063
   iops        : min=  164, max= 8161, avg=7018.54, stdev=1226.90, samples=1063
  lat (usec)   : 50=1.33%, 100=87.71%, 250=10.31%, 500=0.27%, 750=0.13%
  lat (usec)   : 1000=0.09%
  lat (msec)   : 2=0.10%, 4=0.04%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 750=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=4432, avg=46.52, stdev=19.53
    sync percentiles (usec):
     |  1.00th=[   17],  5.00th=[   23], 10.00th=[   30], 20.00th=[   35],
     | 30.00th=[   39], 40.00th=[   41], 50.00th=[   44], 60.00th=[   48],
     | 70.00th=[   52], 80.00th=[   57], 90.00th=[   67], 95.00th=[   77],
     | 99.00th=[   99], 99.50th=[  111], 99.90th=[  151], 99.95th=[  196],
     | 99.99th=[  578]
  cpu          : usr=2.74%, sys=15.41%, ctx=8321846, majf=0, minf=263
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3761722,0,3761720 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=14.3GiB (15.4GB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/7503376, merge=0/1281, ticks=8/429080, in_queue=283612, util=100.00%
s6p6-mx500-4k-sync-write-10: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 10 processes

s6p6-mx500-4k-sync-write-10: (groupid=0, jobs=10): err= 0: pid=1965478: Sat Apr 15 13:59:08 2023
  write: IOPS=66.8k, BW=261MiB/s (274MB/s)(15.3GiB/60001msec); 0 zone resets
    clat (usec): min=39, max=1439.0k, avg=97.93, stdev=2922.74
     lat (usec): min=39, max=1439.0k, avg=98.17, stdev=2922.74
    clat percentiles (usec):
     |  1.00th=[   52],  5.00th=[   59], 10.00th=[   63], 20.00th=[   69],
     | 30.00th=[   73], 40.00th=[   76], 50.00th=[   80], 60.00th=[   84],
     | 70.00th=[   89], 80.00th=[   96], 90.00th=[  112], 95.00th=[  137],
     | 99.00th=[  227], 99.50th=[  338], 99.90th=[ 1254], 99.95th=[ 1762],
     | 99.99th=[ 4621]
   bw (  KiB/s): min= 2840, max=30728, per=10.25%, avg=27388.04, stdev=3766.68, samples=1160
   iops        : min=  710, max= 7682, avg=6846.99, stdev=941.67, samples=1160
  lat (usec)   : 50=0.54%, 100=83.26%, 250=15.42%, 500=0.38%, 750=0.14%
  lat (usec)   : 1000=0.12%
  lat (msec)   : 2=0.10%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=7, max=5540, avg=49.49, stdev=22.06
    sync percentiles (usec):
     |  1.00th=[   18],  5.00th=[   27], 10.00th=[   32], 20.00th=[   37],
     | 30.00th=[   40], 40.00th=[   43], 50.00th=[   47], 60.00th=[   50],
     | 70.00th=[   55], 80.00th=[   61], 90.00th=[   72], 95.00th=[   83],
     | 99.00th=[  109], 99.50th=[  120], 99.90th=[  157], 99.95th=[  198],
     | 99.99th=[  570]
  cpu          : usr=2.54%, sys=15.55%, ctx=8920399, majf=0, minf=306
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4007908,0,4007901 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=261MiB/s (274MB/s), 261MiB/s-261MiB/s (274MB/s-274MB/s), io=15.3GiB (16.4GB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=0/7994125, merge=0/1810, ticks=0/481091, in_queue=314304, util=100.00%
s6p6-mx500-4k-sync-write-11: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 11 processes

s6p6-mx500-4k-sync-write-11: (groupid=0, jobs=11): err= 0: pid=1966875: Sat Apr 15 14:00:09 2023
  write: IOPS=65.9k, BW=257MiB/s (270MB/s)(15.1GiB/60001msec); 0 zone resets
    clat (usec): min=39, max=1686.6k, avg=112.23, stdev=3636.55
     lat (usec): min=39, max=1686.6k, avg=112.46, stdev=3636.55
    clat percentiles (usec):
     |  1.00th=[   56],  5.00th=[   64], 10.00th=[   69], 20.00th=[   74],
     | 30.00th=[   79], 40.00th=[   83], 50.00th=[   87], 60.00th=[   91],
     | 70.00th=[   97], 80.00th=[  105], 90.00th=[  122], 95.00th=[  153],
     | 99.00th=[  269], 99.50th=[  570], 99.90th=[ 1598], 99.95th=[ 2540],
     | 99.99th=[20841]
   bw (  KiB/s): min= 2336, max=29104, per=9.41%, avg=24783.50, stdev=3779.08, samples=1265
   iops        : min=  584, max= 7276, avg=6195.84, stdev=944.78, samples=1265
  lat (usec)   : 50=0.24%, 100=74.87%, 250=23.73%, 500=0.63%, 750=0.14%
  lat (usec)   : 1000=0.16%
  lat (msec)   : 2=0.17%, 4=0.04%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=7, max=2641, avg=52.29, stdev=23.20
    sync percentiles (usec):
     |  1.00th=[   18],  5.00th=[   29], 10.00th=[   34], 20.00th=[   38],
     | 30.00th=[   42], 40.00th=[   45], 50.00th=[   48], 60.00th=[   52],
     | 70.00th=[   58], 80.00th=[   65], 90.00th=[   78], 95.00th=[   91],
     | 99.00th=[  121], 99.50th=[  135], 99.90th=[  182], 99.95th=[  235],
     | 99.99th=[  660]
  cpu          : usr=2.33%, sys=14.89%, ctx=8858792, majf=0, minf=362
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3952376,0,3952370 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=257MiB/s (270MB/s), 257MiB/s-257MiB/s (270MB/s-270MB/s), io=15.1GiB (16.2GB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=16/7885609, merge=0/1997, ticks=3/537055, in_queue=363488, util=99.93%
s6p6-mx500-4k-sync-write-12: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 12 processes

s6p6-mx500-4k-sync-write-12: (groupid=0, jobs=12): err= 0: pid=1968575: Sat Apr 15 14:01:09 2023
  write: IOPS=69.8k, BW=273MiB/s (286MB/s)(15.0GiB/60001msec); 0 zone resets
    clat (usec): min=39, max=1537.6k, avg=113.13, stdev=2603.89
     lat (usec): min=40, max=1537.6k, avg=113.37, stdev=2603.89
    clat percentiles (usec):
     |  1.00th=[   59],  5.00th=[   68], 10.00th=[   73], 20.00th=[   80],
     | 30.00th=[   85], 40.00th=[   89], 50.00th=[   94], 60.00th=[   99],
     | 70.00th=[  105], 80.00th=[  115], 90.00th=[  135], 95.00th=[  172],
     | 99.00th=[  379], 99.50th=[  824], 99.90th=[ 1483], 99.95th=[ 1795],
     | 99.99th=[ 3097]
   bw (  KiB/s): min=18264, max=26888, per=8.55%, avg=23887.21, stdev=1322.33, samples=1392
   iops        : min= 4566, max= 6722, avg=5971.78, stdev=330.58, samples=1392
  lat (usec)   : 50=0.14%, 100=62.14%, 250=35.78%, 500=1.15%, 750=0.18%
  lat (usec)   : 1000=0.31%
  lat (msec)   : 2=0.26%, 4=0.03%, 10=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=7, max=7609, avg=56.35, stdev=29.06
    sync percentiles (usec):
     |  1.00th=[   19],  5.00th=[   31], 10.00th=[   35], 20.00th=[   40],
     | 30.00th=[   44], 40.00th=[   47], 50.00th=[   51], 60.00th=[   56],
     | 70.00th=[   62], 80.00th=[   71], 90.00th=[   86], 95.00th=[  101],
     | 99.00th=[  137], 99.50th=[  153], 99.90th=[  215], 99.95th=[  310],
     | 99.99th=[  971]
  cpu          : usr=2.34%, sys=15.82%, ctx=9473203, majf=0, minf=429
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4190042,0,4190033 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=15.0GiB (17.2GB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/8357765, merge=0/2302, ticks=6/583349, in_queue=379368, util=99.52%
s6p6-mx500-4k-sync-write-13: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 13 processes

s6p6-mx500-4k-sync-write-13: (groupid=0, jobs=13): err= 0: pid=1969933: Sat Apr 15 14:02:10 2023
  write: IOPS=71.2k, BW=278MiB/s (292MB/s)(16.3GiB/60001msec); 0 zone resets
    clat (usec): min=40, max=1470.2k, avg=122.53, stdev=3322.84
     lat (usec): min=41, max=1470.2k, avg=122.77, stdev=3322.84
    clat percentiles (usec):
     |  1.00th=[   62],  5.00th=[   74], 10.00th=[   80], 20.00th=[   87],
     | 30.00th=[   93], 40.00th=[   97], 50.00th=[  102], 60.00th=[  108],
     | 70.00th=[  114], 80.00th=[  123], 90.00th=[  141], 95.00th=[  167],
     | 99.00th=[  293], 99.50th=[  545], 99.90th=[ 1467], 99.95th=[ 1975],
     | 99.99th=[ 4015]
   bw (  KiB/s): min= 3216, max=25512, per=7.89%, avg=22453.91, stdev=3133.68, samples=1509
   iops        : min=  804, max= 6378, avg=5613.45, stdev=783.42, samples=1509
  lat (usec)   : 50=0.07%, 100=45.79%, 250=52.68%, 500=0.93%, 750=0.12%
  lat (usec)   : 1000=0.18%
  lat (msec)   : 2=0.18%, 4=0.04%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=7, max=4601, avg=57.78, stdev=29.67
    sync percentiles (usec):
     |  1.00th=[   20],  5.00th=[   31], 10.00th=[   36], 20.00th=[   40],
     | 30.00th=[   44], 40.00th=[   47], 50.00th=[   51], 60.00th=[   57],
     | 70.00th=[   63], 80.00th=[   73], 90.00th=[   90], 95.00th=[  106],
     | 99.00th=[  145], 99.50th=[  163], 99.90th=[  219], 99.95th=[  293],
     | 99.99th=[ 1029]
  cpu          : usr=2.04%, sys=15.55%, ctx=9712015, majf=0, minf=449
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4271211,0,4271202 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=278MiB/s (292MB/s), 278MiB/s-278MiB/s (292MB/s-292MB/s), io=16.3GiB (17.5GB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/8518627, merge=0/2963, ticks=10/641828, in_queue=427732, util=99.78%
s6p6-mx500-4k-sync-write-14: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 14 processes

s6p6-mx500-4k-sync-write-14: (groupid=0, jobs=14): err= 0: pid=1971297: Sat Apr 15 14:03:11 2023
  write: IOPS=68.0k, BW=266MiB/s (279MB/s)(15.6GiB/60001msec); 0 zone resets
    clat (usec): min=39, max=445362, avg=139.73, stdev=1741.84
     lat (usec): min=40, max=445363, avg=139.95, stdev=1741.84
    clat percentiles (usec):
     |  1.00th=[   65],  5.00th=[   78], 10.00th=[   85], 20.00th=[   94],
     | 30.00th=[  100], 40.00th=[  105], 50.00th=[  112], 60.00th=[  117],
     | 70.00th=[  124], 80.00th=[  135], 90.00th=[  157], 95.00th=[  190],
     | 99.00th=[  281], 99.50th=[  388], 99.90th=[ 1418], 99.95th=[ 2089],
     | 99.99th=[69731]
   bw (  KiB/s): min= 3616, max=23832, per=7.13%, avg=19416.29, stdev=3783.64, samples=1667
   iops        : min=  904, max= 5958, avg=4854.05, stdev=945.91, samples=1667
  lat (usec)   : 50=0.07%, 100=30.31%, 250=67.99%, 500=1.21%, 750=0.10%
  lat (usec)   : 1000=0.16%
  lat (msec)   : 2=0.11%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=7, max=4449, avg=63.82, stdev=32.37
    sync percentiles (usec):
     |  1.00th=[   21],  5.00th=[   33], 10.00th=[   37], 20.00th=[   42],
     | 30.00th=[   47], 40.00th=[   51], 50.00th=[   56], 60.00th=[   62],
     | 70.00th=[   71], 80.00th=[   83], 90.00th=[  103], 95.00th=[  124],
     | 99.00th=[  169], 99.50th=[  190], 99.90th=[  239], 99.95th=[  273],
     | 99.99th=[  619]
  cpu          : usr=1.72%, sys=16.03%, ctx=9390418, majf=0, minf=430
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,4082165,0,4082158 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.7GB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/8140388, merge=0/2538, ticks=6/696530, in_queue=459160, util=98.86%
s6p6-mx500-4k-sync-write-15: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 15 processes

s6p6-mx500-4k-sync-write-15: (groupid=0, jobs=15): err= 0: pid=1973017: Sat Apr 15 14:04:12 2023
  write: IOPS=59.7k, BW=233MiB/s (245MB/s)(13.7GiB/60001msec); 0 zone resets
    clat (usec): min=40, max=2320.8k, avg=181.52, stdev=6375.22
     lat (usec): min=40, max=2320.8k, avg=181.76, stdev=6375.22
    clat percentiles (usec):
     |  1.00th=[    68],  5.00th=[    83], 10.00th=[    91], 20.00th=[   100],
     | 30.00th=[   108], 40.00th=[   114], 50.00th=[   120], 60.00th=[   126],
     | 70.00th=[   133], 80.00th=[   143], 90.00th=[   165], 95.00th=[   200],
     | 99.00th=[   289], 99.50th=[   379], 99.90th=[  1500], 99.95th=[  2737],
     | 99.99th=[130548]
   bw (  KiB/s): min=  448, max=21952, per=7.00%, avg=16729.15, stdev=4107.06, samples=1695
   iops        : min=  112, max= 5488, avg=4182.27, stdev=1026.76, samples=1695
  lat (usec)   : 50=0.05%, 100=19.77%, 250=78.23%, 500=1.55%, 750=0.09%
  lat (usec)   : 1000=0.14%
  lat (msec)   : 2=0.11%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=6, max=6436, avg=67.12, stdev=37.28
    sync percentiles (usec):
     |  1.00th=[   22],  5.00th=[   34], 10.00th=[   38], 20.00th=[   43],
     | 30.00th=[   47], 40.00th=[   52], 50.00th=[   58], 60.00th=[   64],
     | 70.00th=[   74], 80.00th=[   88], 90.00th=[  112], 95.00th=[  135],
     | 99.00th=[  186], 99.50th=[  206], 99.90th=[  265], 99.95th=[  310],
     | 99.99th=[  832]
  cpu          : usr=1.58%, sys=14.03%, ctx=8329821, majf=0, minf=433
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3583466,0,3583455 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=233MiB/s (245MB/s), 233MiB/s-233MiB/s (245MB/s-245MB/s), io=13.7GiB (14.7GB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/7141224, merge=0/3991, ticks=6/766419, in_queue=544528, util=98.94%
s6p6-mx500-4k-sync-write-16: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.12
Starting 16 processes

s6p6-mx500-4k-sync-write-16: (groupid=0, jobs=16): err= 0: pid=1974376: Sat Apr 15 14:05:12 2023
  write: IOPS=61.9k, BW=242MiB/s (254MB/s)(14.2GiB/60001msec); 0 zone resets
    clat (usec): min=40, max=1852.1k, avg=179.28, stdev=4305.96
     lat (usec): min=40, max=1852.1k, avg=179.53, stdev=4305.96
    clat percentiles (usec):
     |  1.00th=[    70],  5.00th=[    86], 10.00th=[    95], 20.00th=[   108],
     | 30.00th=[   116], 40.00th=[   122], 50.00th=[   129], 60.00th=[   137],
     | 70.00th=[   145], 80.00th=[   159], 90.00th=[   190], 95.00th=[   227],
     | 99.00th=[   318], 99.50th=[   490], 99.90th=[  1631], 99.95th=[  3163],
     | 99.99th=[105382]
   bw (  KiB/s): min= 4688, max=20400, per=6.40%, avg=15853.20, stdev=3129.27, samples=1857
   iops        : min= 1172, max= 5100, avg=3963.27, stdev=782.31, samples=1857
  lat (usec)   : 50=0.05%, 100=13.59%, 250=83.18%, 500=2.69%, 750=0.11%
  lat (usec)   : 1000=0.17%
  lat (msec)   : 2=0.13%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
  lat (msec)   : 100=0.01%, 250=0.01%, 500=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=7, max=4757, avg=76.63, stdev=43.66
    sync percentiles (usec):
     |  1.00th=[   24],  5.00th=[   36], 10.00th=[   40], 20.00th=[   46],
     | 30.00th=[   51], 40.00th=[   57], 50.00th=[   64], 60.00th=[   74],
     | 70.00th=[   86], 80.00th=[  103], 90.00th=[  133], 95.00th=[  161],
     | 99.00th=[  221], 99.50th=[  245], 99.90th=[  306], 99.95th=[  343],
     | 99.99th=[  963]
  cpu          : usr=1.41%, sys=16.43%, ctx=8765783, majf=0, minf=516
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,3714739,0,3714728 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=242MiB/s (254MB/s), 242MiB/s-242MiB/s (254MB/s-254MB/s), io=14.2GiB (15.2GB), run=60001-60001msec

Disk stats (read/write):
  sdl: ios=51/7402318, merge=0/6300, ticks=5/804565, in_queue=535444, util=97.64%