When observing the running status of online services on an online server, most people like to use the top command first to see the overall CPU utilization of the current system. For example, for a random machine, the utilization information displayed by the top command is as follows:

This output result is simple to say the least, but not so easy to understand if it is complex. For example:

Question 1: How is the utilization information output by top calculated? Is it accurate?
Question 2: The ni column is nice. It outputs the CPU overhead when processing?
Question 3: wa represents io wait, so is the CPU busy or idle during this period?

Today we have an in-depth study of cpu utilization statistics. Through today's study, you will not only understand the implementation details of CPU utilization statistics, but also have a deeper understanding of indicators such as nice and io wait.

Today we start with our own thoughts!

1. Think about it first

Leaving aside the implementation of Linux, if you have the following requirements, there is a quad-core server with four processes running on it.

Allows you to design and calculate the CPU utilization of the entire system. It supports output like the top command and meets the following requirements:

• The cpu usage rate should be as accurate as possible;
• It is necessary to reflect the instantaneous CPU status at the second level as much as possible.

You can stop and think for a few minutes.

Okay, end of thinking. After thinking about it, you will find that this seemingly simple requirement is actually a bit complicated.

One idea is to add up the execution time of all processes and then divide it by the total system execution time * 4.

There is no problem with this idea. It is possible to use this method to count CPU utilization over a long period of time, and the statistics are accurate enough.

But as long as you have used top, you will know that the cpu utilization output by top is not constant for a long time, but will be dynamically updated in units of 3 seconds by default (this time interval can be set using -d). Our solution can reflect the total utilization, but it is difficult to reflect this instantaneous state. You may think that I can count it as one every 3 seconds, right? But at what point does this 3-second period begin. The granularity is difficult to control.

The core of the previous thinking question is how to solve instantaneous problems. When it comes to the transient state, you may have another idea. Then I will use instant sampling to see how many cores are currently busy. If two of the four cores are busy, the utilization is 50%.

This line of thinking is also correct, but there are two problems:

• The numbers you calculate are all multiples of 25%;
• This instantaneous value can cause wild swings in the CPU usage display.

For example, the picture below:

From the instantaneous state of t1, the system's CPU utilization is undoubtedly 100%, but from the perspective of t2, the usage has become 0%. The idea is in the right direction, but obviously this crude calculation cannot work as elegantly as the top command.

Let’s improve it and combine the above two ideas, maybe we can solve our problem. In terms of sampling, we set the period to be finer, but in terms of calculation, we set the period to be coarser.

We introduce the concept of adoption period, timing, such as sampling every 1 millisecond. If the CPU is running at the moment of sampling, this 1 ms is recorded as used. At this time, an instantaneous CPU usage will be obtained and saved.

When counting the CPU usage within 3 seconds, such as the t1 and t2 time range in the above figure. Then add all the instantaneous values ??during this period and take an average. This can solve the above problem, the statistics are relatively accurate, and the problem of instantaneous values ??oscillating violently and being too coarse-grained (can only change in units of 25%) is avoided.

Some students may ask, what if the CPU changes between two samplings, as shown in the picture below.

When the current sampling point arrives, process A has just finished executing. For a little while, it has not been counted by the previous sampling point, nor can it be counted this time. For process B, it actually only started for a short period of time. It seems a bit too much to record all 1 ms.

This problem does exist, but because our sampling is once every 1 ms, and when we actually check and use it, it is at least on the second level, which will include information from thousands of sampling points, so this error It will not affect our grasp of the overall situation.

In fact, this is how Linux counts system CPU utilization. Although there may be errors, it is enough to be used as a statistical data. In terms of implementation, Linux accumulates all instantaneous values ??into a certain data, rather than actually storing many copies of instantaneous data.

Next, let us enter Linux to see its specific implementation of system cpu utilization statistics.

2. Where is the data used by the top command

The implementation of Linux we mentioned in the previous section is to accumulate instantaneous values ????to a certain data. This value is exposed to the user mode by the kernel through the /proc/stat pseudo file. Linux uses it when calculating system CPU utilization.

Overall, the internal details of the top command work are shown in the figure below.

The top command accesses /proc/stat to obtain various cpu utilization values;

• The kernel calls the stat_open function to handle access to /proc/stat;

• The data accessed by the kernel comes from the kernel_cpustat array and is summarized;

• Print output to user mode.

Next, let’s take a look at each step in detail.

By using strace to trace the various system calls of the top command, you can see its calls to the file.

#?strace?top
...
openat(AT_FDCWD,?"/proc/stat",?O_RDONLY)?=?4
openat(AT_FDCWD,?"/proc/2351514/stat",?O_RDONLY)?=?8
openat(AT_FDCWD,?"/proc/2393539/stat",?O_RDONLY)?=?8
...

“

In addition to /proc/stat, there is also /proc/{pid}/stat broken down by each process, which is used to calculate the cpu utilization of each process.

”

The kernel defines processing functions for each pseudo-file. The processing method of /proc/stat file is proc_stat_operations.

//file:fs/proc/stat.c
static?int?__init?proc_stat_init(void)
{
?proc_create("stat",?0,?NULL,?&proc_stat_operations);
?return?0;
}

static?const?struct?file_operations?proc_stat_operations?=?{
?.open??=?stat_open,
?...
};

proc_stat_operations contains the operation methods corresponding to this file. When the /proc/stat file is opened, stat_open will be called. stat_open calls single_open_size and show_stat in sequence to output the data content. Let’s take a look at its code:

//file:fs/proc/stat.c
static?int?show_stat(struct?seq_file?*p,?void?*v)
{
?u64?user,?nice,?system,?idle,?iowait,?irq,?softirq,?steal;

?for_each_possible_cpu(i)?{
??struct?kernel_cpustat?*kcs?=?&kcpustat_cpu(i);

??user?+=?kcs->cpustat[CPUTIME_USER];
??nice?+=?kcs->cpustat[CPUTIME_NICE];
??system?+=?kcs->cpustat[CPUTIME_SYSTEM];
??idle?+=?get_idle_time(kcs,?i);
??iowait?+=?get_iowait_time(kcs,?i);
??irq?+=?kcs->cpustat[CPUTIME_IRQ];
??softirq?+=?kcs->cpustat[CPUTIME_SOFTIRQ];
??...
?}

?//轉(zhuǎn)換成節(jié)拍數(shù)并打印出來
?seq_put_decimal_ull(p,?"cpu??",?nsec_to_clock_t(user));
?seq_put_decimal_ull(p,?"?",?nsec_to_clock_t(nice));
?seq_put_decimal_ull(p,?"?",?nsec_to_clock_t(system));
?seq_put_decimal_ull(p,?"?",?nsec_to_clock_t(idle));
?seq_put_decimal_ull(p,?"?",?nsec_to_clock_t(iowait));
?seq_put_decimal_ull(p,?"?",?nsec_to_clock_t(irq));
?seq_put_decimal_ull(p,?"?",?nsec_to_clock_t(softirq));
?...
}

In the above code, for_each_possible_cpu is traversing the kcpustat_cpu variable that stores cpu usage data. This variable is a percpu variable, which prepares an array element for each logical core. It stores various events corresponding to the current core, including user, nice, system, idel, iowait, irq, softirq, etc.

In this loop, add up each usage of each core. Finally, the data is output through seq_put_decimal_ull.

Note that in the kernel, each time is actually recorded in nanoseconds, but they are all converted into beat units when output. As for the length of the beat unit, we will introduce it in the next section. In short, the output of /proc/stat is read from the percpu variable kernel_cpustat.

Let’s take a look at when the data in this variable was added.

三、統(tǒng)計數(shù)據(jù)怎么來的

前面我們提到內(nèi)核是以采樣的方式來統(tǒng)計 cpu 使用率的。這個采樣周期依賴的是 Linux 時間子系統(tǒng)中的定時器。

Linux 內(nèi)核每隔固定周期會發(fā)出 timer interrupt (IRQ 0)，這有點像樂譜中的節(jié)拍的概念。每隔一段時間，就打出一個拍子，Linux 就響應(yīng)之并處理一些事情。

一個節(jié)拍的長度是多長時間，是通過 CONFIG_HZ 來定義的。它定義的方式是每一秒有幾次 timer interrupts。不同的系統(tǒng)中這個節(jié)拍的大小可能不同，通常在 1 ms 到 10 ms 之間。可以在自己的 Linux config 文件中找到它的配置。

#?grep?^CONFIG_HZ?/boot/config-5.4.56.bsk.10-amd64
CONFIG_HZ=1000

從上述結(jié)果中可以看出，我的機器每秒要打出 1000 次節(jié)拍。也就是每 1 ms 一次。

每次當時間中斷到來的時候，都會調(diào)用 update_process_times 來更新系統(tǒng)時間。更新后的時間都存儲在我們前面提到的 percpu 變量 kcpustat_cpu 中。

我們來詳細看下匯總過程 update_process_times 的源碼，它位于 kernel/time/timer.c 文件中。

//file:kernel/time/timer.c
void?update_process_times(int?user_tick)
{
?struct?task_struct?*p?=?current;

?//進行時間累積處理
?account_process_tick(p,?user_tick);
?...
}

這個函數(shù)的參數(shù) user_tick 指的是采樣的瞬間是處于內(nèi)核態(tài)還是用戶態(tài)。接下來調(diào)用 account_process_tick。

//file:kernel/sched/cputime.c
void?account_process_tick(struct?task_struct?*p,?int?user_tick)
{
?cputime?=?TICK_NSEC;
?...

?if?(user_tick)
??//3.1?統(tǒng)計用戶態(tài)時間
??account_user_time(p,?cputime);
?else?if?((p?!=?rq->idle)?||?(irq_count()?!=?HARDIRQ_OFFSET))
??//3.2?統(tǒng)計內(nèi)核態(tài)時間
??account_system_time(p,?HARDIRQ_OFFSET,?cputime);
?else
??//3.3?統(tǒng)計空閑時間
??account_idle_time(cputime);
}

在這個函數(shù)中，首先設(shè)置 cputime = TICK_NSEC, 一個 TICK_NSEC 的定義是一個節(jié)拍所占的納秒數(shù)。接下來根據(jù)判斷結(jié)果分別執(zhí)行 account_user_time、account_system_time 和 account_idle_time 來統(tǒng)計用戶態(tài)、內(nèi)核態(tài)和空閑時間。

3.1 用戶態(tài)時間統(tǒng)計

//file:kernel/sched/cputime.c
void?account_user_time(struct?task_struct?*p,?u64?cputime)
{
?//分兩種種情況統(tǒng)計用戶態(tài)?CPU?的使用情況
?int?index;
?index?=?(task_nice(p)?>?0)???CPUTIME_NICE?:?CPUTIME_USER;

?//將時間累積到?/proc/stat?中
?task_group_account_field(p,?index,?cputime);
?......
}

account_user_time 函數(shù)主要分兩種情況統(tǒng)計：

• 如果進程的 nice 值大于 0，那么將會增加到 CPU 統(tǒng)計結(jié)構(gòu)的 nice 字段中。
• 如果進程的 nice 值小于等于 0，那么增加到 CPU 統(tǒng)計結(jié)構(gòu)的 user 字段中。

看到這里，開篇的問題 2 就有答案了，其實用戶態(tài)的時間不只是 user 字段，nice 也是。之所以要把 nice 分出來，是為了讓 Linux 用戶更一目了然地看到調(diào)過 nice 的進程所占的 cpu 周期有多少。

我們平時如果想要觀察系統(tǒng)的用戶態(tài)消耗的時間的話，應(yīng)該是將 top 中輸出的 user 和 nice 加起來一并考慮，而不是只看 user！

接著調(diào)用 task_group_account_field 來把時間加到前面我們用到的 kernel_cpustat 內(nèi)核變量中。

//file:kernel/sched/cputime.c
static?inline?void?task_group_account_field(struct?task_struct?*p,?int?index,
??????u64?tmp)
{
?__this_cpu_add(kernel_cpustat.cpustat[index],?tmp);
?...
}

3.2 內(nèi)核態(tài)時間統(tǒng)計

我們再來看內(nèi)核態(tài)時間是如何統(tǒng)計的，找到 account_system_time 的代碼。

//file:kernel/sched/cputime.c
void?account_system_time(struct?task_struct?*p,?int?hardirq_offset,?u64?cputime)
{
?if?(hardirq_count()?-?hardirq_offset)
??index?=?CPUTIME_IRQ;
?else?if?(in_serving_softirq())
??index?=?CPUTIME_SOFTIRQ;
?else
??index?=?CPUTIME_SYSTEM;

?account_system_index_time(p,?cputime,?index);
}

內(nèi)核態(tài)的時間主要分 3 種情況進行統(tǒng)計。

• 如果當前處于硬中斷執(zhí)行上下文, 那么統(tǒng)計到 irq 字段中；
• 如果當前處于軟中斷執(zhí)行上下文, 那么統(tǒng)計到 softirq 字段中；
• 否則統(tǒng)計到 system 字段中。

判斷好要加到哪個統(tǒng)計項中后，依次調(diào)用 account_system_index_time、task_group_account_field 來將這段時間加到內(nèi)核變量 kernel_cpustat 中。

//file:kernel/sched/cputime.c
static?inline?void?task_group_account_field(struct?task_struct?*p,?int?index,
??????u64?tmp)
{?
?__this_cpu_add(kernel_cpustat.cpustat[index],?tmp);
}

3.3 空閑時間的累積

沒錯，在內(nèi)核變量 kernel_cpustat 中不僅僅是統(tǒng)計了各種用戶態(tài)、內(nèi)核態(tài)的使用時間，空閑也一并統(tǒng)計起來了。

如果在采樣的瞬間，cpu 既不在內(nèi)核態(tài)也不在用戶態(tài)的話，就將當前節(jié)拍的時間都累加到 idle 中。

//file:kernel/sched/cputime.c
void?account_idle_time(u64?cputime)
{
?u64?*cpustat?=?kcpustat_this_cpu->cpustat;
?struct?rq?*rq?=?this_rq();

?if?(atomic_read(&rq->nr_iowait)?>?0)
??cpustat[CPUTIME_IOWAIT]?+=?cputime;
?else
??cpustat[CPUTIME_IDLE]?+=?cputime;
}

在 cpu 空閑的情況下，進一步判斷當前是不是在等待 IO（例如磁盤 IO），如果是的話這段空閑時間會加到 iowait 中，否則就加到 idle 中。從這里，我們可以看到 iowait 其實是 cpu 的空閑時間，只不過是在等待 IO 完成而已。

看到這里，開篇問題 3 也有非常明確的答案了，io wait 其實是 cpu 在空閑狀態(tài)的一項統(tǒng)計，只不過這種狀態(tài)和 idle 的區(qū)別是 cpu 是因為等待 io 而空閑。

四、總結(jié)

本文深入分析了 Linux 統(tǒng)計系統(tǒng) CPU 利用率的內(nèi)部原理。全文的內(nèi)容可以用如下一張圖來匯總：

Linux 中的定時器會以某個固定節(jié)拍，比如 1 ms 一次采樣各個 cpu 核的使用情況，然后將當前節(jié)拍的所有時間都累加到 user/nice/system/irq/softirq/io_wait/idle 中的某一項上。

top 命令是讀取的 /proc/stat 中輸出的 cpu 各項利用率數(shù)據(jù)，而這個數(shù)據(jù)在內(nèi)核中是根據(jù) kernel_cpustat 來匯總并輸出的。

回到開篇問題 1，top 輸出的利用率信息是如何計算出來的，它精確嗎？

/proc/stat 文件輸出的是某個時間點的各個指標所占用的節(jié)拍數(shù)。如果想像 top 那樣輸出一個百分比，計算過程是分兩個時間點 t1, t2 分別獲取一下 stat 文件中的相關(guān)輸出，然后經(jīng)過個簡單的算術(shù)運算便可以算出當前的 cpu 利用率。

再說是否精確。這個統(tǒng)計方法是采樣的，只要是采樣，肯定就不是百分之百精確。但由于我們查看 cpu 使用率的時候往往都是計算 1 秒甚至更長一段時間的使用情況，這其中會包含很多采樣點，所以查看整體情況是問題不大的。

另外從本文，我們也學(xué)到了 top 中輸出的 cpu 時間項目其實大致可以分為三類：

第****一類：用戶態(tài)消耗時間，包括 user 和 nice。如果想看用戶態(tài)的消耗，要將 user 和 nice 加起來看才對。

第二類：內(nèi)核態(tài)消耗時間，包括 irq、softirq 和 system。

第三類：空閑時間，包括 io_wait 和 idle。其中 io_wait 也是 cpu 的空閑狀態(tài)，只不過是在等 io 完成而已。如果只是想看 cpu 到底有多閑，應(yīng)該把 io_wait 和 idle 加起來才對。

The above is the detailed content of How is CPU utilization calculated in Linux?. For more information, please follow other related articles on the PHP Chinese website!