Use iostat to check %util and await for disk saturation; 2. Run iotop to identify high-I/O processes; 3. Check %wa in top or vmstat for CPU I/O wait; 4. Review mount options like noatime and discard, and verify filesystem health; 5. Identify disk type via lsblk -d -o name,rota to optimize accordingly; 6. Tune I/O scheduler: deadline for HDDs, none/mq-deadline for SSDs; 7. Assess RAID, LVM, or virtualization overhead such as thin provisioning or non-virtio disks; 8. Benchmark with dd or fio to test raw performance; 9. Ensure sufficient RAM and adjust vm.dirty_ratio and vm.dirty_background_ratio to manage writeback pressure; addressing these areas systematically resolves most slow disk I/O issues without requiring hardware upgrades.

Slow disk I/O on Linux systems can severely impact application performance, system responsiveness, and overall user experience. Diagnosing and resolving these issues requires a methodical approach to identify bottlenecks in storage, configuration, or system load. Below are key steps and tools to help troubleshoot and improve disk I/O performance.

1. Check Current I/O Utilization with iostat

The first step is to assess whether the disk is actually the bottleneck.

• Install sysstat if not already available:

  

  sudo apt install sysstat    # Debian/Ubuntu
  sudo yum install sysstat    # RHEL/CentOS

• Run iostat to monitor disk activity:

  iostat -x 2 5

  This gives extended statistics every 2 seconds, 5 times.

  

Key metrics to watch:

• %util – Percentage of time the device is busy. Sustained values near 100% indicate saturation.
• await – Average wait time (in ms) for I/O requests. High values suggest congestion.
• svctm – Average service time (deprecated in newer kernels, use cautiously).
• r/s, w/s – Read/write operations per second.

If %util is high and await is increasing, the disk is likely overloaded.

2. Identify Processes Causing High I/O with iotop

Just like top for CPU, iotop shows which processes are generating the most I/O.

• Install and run:

  sudo apt install iotop
  sudo iotop -o

  The -o flag shows only processes currently doing I/O.

Look for:

• Processes with high DISK WRITE or DISK READ.
• Unexpected or runaway processes (e.g., loggers, backups, databases).

Tip: Press r to sort by read, w by write, and p to toggle processes only.

3. Check for I/O Wait in CPU Usage

High %wa (I/O wait) in CPU usage indicates the CPU is idle waiting for disk responses.

• Use top or htop:
  • In top, look at the CPU line: %us, %sy, %ni, %id, %wa
  • If %wa is consistently high (e.g., >20%), I/O is a bottleneck.

Alternatively, use:

vmstat 2 5

Check the wa column under CPU.

4. Evaluate Filesystem and Mount Options

Misconfigured filesystems can severely limit performance.

• Check current mount options:

  mount | grep " / "

  or

  cat /proc/mounts

Common performance-related options:

• noatime or relatime – Prevents updating access time on every read (reduces writes).
• data=ordered (ext4 default) – Good balance; avoid data=journal unless required.
• For SSDs: ensure discard is set (or use periodic fstrim) if using TRIM.

If atime is enabled, consider switching to relatime in /etc/fstab:

/dev/sda1 / ext4 defaults,relatime 0 1

Also, check for filesystem errors or full disks:

df -h              # Check free space
dmesg | grep -i "I/O error"  # Look for hardware errors

5. Assess Disk Type and Hardware Limitations

Not all disks perform the same:

• HDDs are slow for random I/O; optimize for sequential access.
• SSDs/NVMe offer high IOPS but can degrade if over-provisioned or worn.

Check your disk type:

lsblk -d -o name,rota

• rota=1 = rotational (HDD)
• rota=0 = non-rotational (SSD/NVMe)

HDDs benefit from:

• Larger I/O queues
• Sequential access patterns
• Reducing random I/O via caching or batching

6. Tune I/O Scheduler

Linux uses different I/O schedulers (e.g., cfq, deadline, noop, mq-deadline, kyber).

• Check current scheduler:

  cat /sys/block/sda/queue/scheduler

  (Replace sda with your disk)

Recommended:

• For SSD/NVMe: none (via mq-deadline or kyber on modern kernels)
• For HDD: mq-deadline or deadline

Change temporarily:

echo deadline | sudo tee /sys/block/sda/queue/scheduler

Make permanent via kernel boot parameter (e.g., in GRUB):

elevator=deadline

7. Check for RAID, LVM, or Virtualization Overhead

• LVM: Thin provisioning or snapshots can slow I/O.
• RAID arrays: Rebuilds or degraded arrays cause slowness.
• Virtual machines: Host storage contention or using emulated disks (e.g., IDE instead of virtio).

In VMs:

• Use virtio block devices.
• Ensure the host isn’t overcommitted on storage.

8. Test Raw Disk Performance

Use tools to benchmark actual disk speed.

• Sequential write test:

  dd if=/dev/zero of=/tmp/testfile bs=1G count=1 oflag=direct

  (oflag=direct bypasses cache)

• Read test:

  dd if=/tmp/testfile of=/dev/null bs=1G iflag=direct

Better tool: fio (Flexible I/O Tester)

fio --name=randwrite --ioengine=libaio --direct=1 \
    --size=1G --bs=4k --rw=randwrite --runtime=60

Compare results with expected disk specs.

9. Consider Memory and Cache Pressure

Low available memory forces more frequent disk writes (dirty pages) and fewer read caches.

• Check memory usage:

  free -h

• Monitor dirty memory:

  cat /proc/vmstat | grep -E "dirty|writeback"

Tune kernel dirty page settings in /etc/sysctl.conf:

vm.dirty_background_ratio = 5    # Start writing at 5% dirty memory
vm.dirty_ratio = 10              # Throttle at 10%

Avoid overly aggressive flushing that causes I/O bursts.

Summary Checklist

• ? Use iostat and iotop to confirm I/O bottleneck
• ? Check for high %wa in CPU usage
• ? Review mount options (noatime, discard)
• ? Verify disk type and health
• ? Tune I/O scheduler based on storage
• ? Rule out LVM, RAID, or virtualization overhead
• ? Benchmark with fio or dd
• ? Ensure sufficient RAM and tune dirty page settings

Slow disk I/O is rarely a single-point failure — it’s often a mix of workload, configuration, and hardware. Start with monitoring, isolate the cause, and apply targeted fixes. Most issues can be resolved without hardware upgrades if caught early.

The above is the detailed content of Troubleshooting Slow Disk I/O on Linux Systems. For more information, please follow other related articles on the PHP Chinese website!