从/proc/stat文件中取得系统运行信息
2019-04-30 16:25:26 阿炯

/proc文件系统是一个伪文件系统,它只存在内存当中,不占用外存空间,它以文件系统的方式为内核与进程提供通信的接口。用户和应用程序可以通过/proc得到系统的信息,并可以改变内核的某些参数。由于系统的信息,如进程等是动态改变的,所以用户或应用程序读取/proc目录中的文件时,proc文件系统是动态从系统内核读出所需信息并提交的。/proc目录中有一些以数字命名的目录,它们是进程目录。系统中当前运行的每一个进程在/proc下都对应一个以进程号为目录名的目录/proc/pid,为读取进程信息的接口。

/proc/stat文件包含了所有系统进程整体CPU活动的信息,包括CPU运行情况、中断情况、启动时间、上线文切换次数,系统进程整体活动的信息,包括CPU运行情况、中断情况、启动时间、上线文切换次数、运行中的进程等信息都在其中,其中的所有值都是从系统启动开始累计到当前时刻。记录的是的统计信息,不同内核版本中该文件的格式可能不大一致,以下通过实例来说明数据该文件中各字段的含义。只针对Linux内核2.6.24-24版本之上。

# more /proc/stat
cpu  6980 48 10294 4914552 603 2 1629 0 0 0
cpu0 1939 15 3134 2468373 315 0 348 0 0 0
cpu1 5040 32 7159 2446178 287 2 1281 0 0 0
intr 2703995 156 50 0 0 0 0 0 0 8 0 0 0 204 0 0 0 215521 0 1 382055 780 79174 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 4048544
btime 1556582042
processes 1813
procs_running 1
procs_blocked 0
softirq 3481902 1 1352575 10294 584970 77734 0 2272 937412 24544 492100

第一行的数值表示的是CPU总的使用情况,所以我们只要用第一行的数字计算就可以了,下面解析第一行各数值的含义:

参数    解析(单位:jiffies)
(jiffies是内核中的一个全局变量,用来记录自系统启动一来产生的节拍数,在Linux中,一个节拍大致可理解为操作系统进程调度的最小时间片,不同linux内核可能值有不同,通常在1ms到10ms之间。sysconf(_SC_CLK_TCK)一般地定义为jiffies(一般地等于10ms))
user (6980)    从系统启动开始累计到当前时刻,处于用户态的运行时间,不包含 nice值为负进程。
nice (48)      从系统启动开始累计到当前时刻,nice值为负的进程所占用的CPU时间
system (10294)  从系统启动开始累计到当前时刻,处于核心态的运行时间
idle (4914552)   从系统启动开始累计到当前时刻,除IO等待时间以外的其它等待时间
iowait (603) 从系统启动开始累计到当前时刻,IO等待时间(since 2.5.41)
irq (2)           从系统启动开始累计到当前时刻,硬中断时间(since 2.6.0-test4)
softirq(1629)      从系统启动开始累计到当前时刻,软中断时间(since 2.6.0-test4)stealstolen(0), which is the time spent in other operating systems when running in a virtualized environment(since 2.6.11)
stealstolen(0)     which is the time spent in other operating systems when running in a virtualized environment(since 2.6.11)
guest(0)        which is the time spent running a virtual  CPU  for  guest operating systems under the control of the Linux kernel(since 2.6.24)

结论:总的cpu时间totalCpuTime = user + nice + system + idle + iowait + irq + softirq + stealstolen + guest

iowait时间是不可靠值,理由如下:CPU不会等待I/O执行完成,而iowait是等待I/O完成的时间。当CPU进入idle状态,很可能会调度另一个task执行,所以iowait计算时间偏小;在多核CPU情况下,iowait的计算并非某一个核,因此计算每一个cpu的iowait非常困难。具体可查阅相关资料

其它行说明

intr:系统启动以来的所有interrupts的次数情况
ctxt: 系统上下文切换次数
btime:启动时长(单位:秒),从Epoch(即1970零时)开始到系统启动所经过的时长,每次启动会改变。此处指为1556582042,转换北京时间为2019/4/30 7:54:2
processes:系统启动后所创建过的进程数量。当短时间该值特别大,系统可能出现异常
procs_running:处于Runnable状态的进程个数
procs_blocked:处于等待I/O完成的进程个数

Since Linux 2.6.11, there is an 8th column called 'steal' - counts the ticks spent executing other virtual hosts (in virtualised environments like Xen)
Since Linux 2.6.24, there is a 9th column called 'guest' - counts the time spent running a virtual CPU for guest operating systems under the control of the Linux kernel
Since Linux 2.6.33, there is a 10th column called 'guest_nice' - Time spent running a niced guest (virtual CPU for guest operating systems under the control of the Linux kernel).

The "procs_running" line gives the number of processes currently running on CPUs (Linux 2.5.45 onwards)
The "procs_blocked" line gives the number of processes currently blocked, waiting for I/O to complete (Linux 2.5.45 onwards)

/proc/<pid>/stat

该文件包含了某一进程所有的活动的信息,该文件中的所有值都是从系统启动开始累计到当前时刻。以下通过实例数据来说明该文件中各字段的含义。
 
# more /proc/6873/stat

6873 (freeoa.bin) R 6723 6873 6723 34819 6873 8388608 77 0 0 0 41958 31 0 0 25 0 3 0 5882654 1409024 56 4294967295 134512640 134513720 3215579040 0 2097798 0 0 0 0 0 0 0 17 0 0 0

说明:以下只解释对计算Cpu使用率有用相关参数。

参数    解释
pid=6873    进程号
utime=1587    该任务在用户态运行的时间,单位为jiffies
stime=41958    该任务在核心态运行的时间,单位为jiffies
cutime=0    所有已死线程在用户态运行的时间,单位为jiffies
cstime=0    所有已死在核心态运行的时间,单位为jiffies

结论:进程的总Cpu时间processCpuTime = utime + stime + cutime + cstime,该值包括其所有线程的cpu时间。

其它所有参数
# more /proc/8385/stat
1557 (system_server) S 823 823 0 0 -1 1077952832 //1~9
2085481 15248 2003 27 166114 129684 26 30  //10~17
10 -10 221 0 2284 2790821888 93087 18446744073709551615 //18~25
1 1 0 0 0 0 6660 0 36088 0 0 0 17 3 0 0 0 0 0 0 0 0 0 0 0 0 0

pid:进程ID.
comm:task_struct结构体的进程名
state:进程状态, 此处为S
ppid:父进程ID(父进程是指通过fork方式,通过clone并非父进程)
pgrp:进程组ID
session:进程会话组ID
tty_nr:当前进程的tty终点设备号
tpgid:控制进程终端的前台进程号
flags:进程标识位,定义在include/linux/sched.h中的PF_*, 此处等于1077952832
minflt:次要缺页中断的次数,即无需从磁盘加载内存页. 比如COW和匿名页
cminflt:当前进程等待子进程的minflt
majflt:主要缺页中断的次数,需要从磁盘加载内存页. 比如map文件
majflt:当前进程等待子进程的majflt
utime:该进程处于用户态的时间,单位jiffies,此处等于166114
stime:该进程处于内核态的时间,单位jiffies,此处等于129684
cutime:当前进程等待子进程的utime
cstime:当前进程等待子进程的utime
priority:进程优先级, 此次等于10.
nice:nice值,取值范围[19, -20],此处等于-10
num_threads:线程个数, 此处等于221
itrealvalue:该字段已废弃,恒等于0
starttime:自系统启动后的进程创建时间,单位jiffies,此处等于2284
vsize:进程的虚拟内存大小,单位为bytes
rss:进程独占内存+共享库,单位pages,此处等于93087
rsslim:rss大小上限

说明:
第10~17行主要是随着时间而改变的量;
内核时间单位,sysconf(_SC_CLK_TCK)一般地定义为jiffies(一般地等于10ms)
starttime:此值单位为jiffies, 结合/proc/stat的btime,可知道每一个线程启动的时间点
1500827856 + 2284/100 = 1500827856, 转换成北京时间为2017/7/24 0:37:58

第四行数据很少使用,只说一下该行第7至9个数的含义:
signal:即将要处理的信号,十进制,此处等于6660
blocked:阻塞的信号,十进制
sigignore:被忽略的信号,十进制,此处等于36088

/proc/<pid>/task/<tid>/stat

该文件包含了某一进程所有的活动的信息,该文件中的所有值都是从系统启动开始累计到当前时刻。该文件的内容格式以及各字段的含义同/proc/<pid>/stat文件。

注意:该文件中的tid字段表示的不再是进程号,而是linux中的轻量级进程(lwp),即我们通常所说的线程。

结论:线程Cpu时间threadCpuTime = utime + stime

单核情况下Cpu使用率的计算

通过读取/proc/stat、/proc/<pid>/stat、/proc/<pid>/task/<tid>/stat以及/proc/cpuinfo这几个文件获取总的Cpu时间、进程的Cpu时间、线程的Cpu时间以及Cpu的个数的信息,然后通过一定的算法进行计算(采样两个足够短的时间间隔的Cpu快照与进程快照来计算进程的Cpu使用率)。

总的Cpu使用率计算

计算方法:
1、采样两个足够短的时间间隔的Cpu快照,分别记作t1,t2,其中t1、t2的结构均为:
(user、nice、system、idle、iowait、irq、softirq、stealstolen、guest)的9元组;

2、计算总的Cpu时间片totalCpuTime
a)把第一次的所有cpu使用情况求和,得到s1;
b)把第二次的所有cpu使用情况求和,得到s2;
c)s2 - s1得到这个时间间隔内的所有时间片,即totalCpuTime = j2 - j1 ;

3、计算空闲时间idle
idle对应第四列的数据,用第二次的第四列 - 第一次的第四列即可
idle=第二次的第四列 - 第一次的第四列

4、计算cpu使用率
pcpu =100* (total-idle)/total

某进程cpu使用率的计算
1.采样两个足够短的时间间隔的cpu快照与进程快照
a)每一个cpu快照均为(user、nice、system、idle、iowait、irq、softirq、stealstolen、guest)的9元组;
b)每一个进程快照均为(utime、stime、cutime、cstime)的4元组;

2.分别根据结论2、结论3计算出两个时刻的总的cpu时间与进程的cpu时间,分别记作:totalCpuTime1、totalCpuTime2、processCpuTime1、processCpuTime2

3.计算该进程的cpu使用率pcpu = 100*( processCpuTime2 – processCpuTime1) / (totalCpuTime2 – totalCpuTime1) (按100%计算,如果是多核情况下还需乘以cpu的个数);

某线程cpu使用率的计算
1.采样两个足够短的时间隔的cpu快照与线程快照
a)每一个cpu快照均为(user、nice、system、idle、iowait、irq、softirq、stealstealon、guest)的9元组;
b)每一个线程快照均为 (utime、stime)的2元组;

2.分别根据结论2、结论4计算出两个时刻的总的cpu时间与线程的cpu时间,分别记作:totalCpuTime1、totalCpuTime2、threadCpuTime1、threadCpuTime2

3.计算该线程的cpu使用率pcpu = 100*( threadCpuTime2 – threadCpuTime1) / (totalCpuTime2 – totalCpuTime1) (按100%计算,如果是多核情况下还需乘以cpu的个数);

多核情况下cpu使用率的计算  

以下通过数据来说明多核情况下某一进程cpu使用率是按cpu个数*100%计算的。第一组数据是通过ps  -eLo pid,lwp,pcpu | grep 9140命令查看相关进程号中各线程的详细信息。第二组数据是通过 ps命令查看相关进程号的cpu使用率。

可能的问题:
1.不同内核版本/proc/stat文件格式不大一致,/proc/stat文件中第一行为总的cpu使用情况。
各个版本都有的4个字段:user、nice、system、idle

2.5.41版本新增字段:iowait
2.6.0-test4新增字段:irq、softirq
2.6.11新增字段:stealstolen:which is the time spent in other operating systems when running in a virtualized environment
2.6.24新增字段:guest:which is the time spent running a virtual  CPU  for  guest operating systems under the control of the Linux kernel

2./proc/pid/task目录是Linux 2.6.0-test6之后才有的功能。

3.关于出现cpu使用率为负的情况,目前想到的解决方案是如果出现负值,连续采样计算cpu使用率直到为非负。

4.有些线程生命周期较短,可能在我们采样期间就已经退出了。

关于/proc/uptime

more /proc/uptime
25591.03 50518.07

第一个值代表从开机到现在的累积时间(单位为秒),开机后运行82044秒;第二个值代表从开机到现在的CPU空闲时间,单位为秒。

小技巧:结合btime(/proc/stat)获取当前的绝对时间1556582042 + 25591 = 1556607633,转换成北京时间2019/4/30 15:0:33,也就是当前执行more命令的时间点。

小总结

/proc/stat
该文件包含了所有CPU活动的信息,该文件中的所有值都是从系统启动开始累计到当前时刻。不同内核版本(2.6.24以上)中该文件的格式可能不大一致。

/proc/<pid>/stat
该文件包含了某一进程所有的活动的信息,该文件中的所有值都是从系统启动开始累计到当前时刻。

/proc/<pid>/task/<tid>/stat
该文件包含了某一进程所有的活动的信息,该文件中的所有值都是从系统启动开始累计到当前时刻。该文件的内容格式以及各字段的含义同/proc/<pid>/stat文件。注意该文件中的tid字段表示的不再是进程号,而是linux中的轻量级进程(lwp),即通常所说的线程。线程Cpu时间threadCpuTime = utime + stime。


通过ps命令可以查看系统中相关进程的Cpu使用率的信息,下为linux man文档中对ps命令输出中有关Cpu使用率的解释:
CPU usage is currently expressed as the percentage of time spent running during the entire lifetime of a process. This is not ideal, and it does not conform to the standards that ps otherwise conforms to. CPU usage is unlikely to add up to exactly 100%.

%cpu   cpu utilization of the process in "##.#" format. It is the CPU time used divided by the time the process has been running (cputime/realtime ratio), expressed as a percentage. It will not add up to 100% unless you are lucky.

ps命令算出来的cpu使用率相对于进程启动时的平均值,随着进程运行时间的增大,该值会趋向于平缓。

通过top命令可以查看系统中相关进程的实时信息(cpu使用率等),下为man文档中对top命令输出中有关进程cpu使用率的解释:
#C  --  Last used CPU (SMP)    A number representing the last used processor. In a true  SMP  environment  this  will  likely change  frequently  since  the  kernel intentionally uses weak affinity.  Also, the very act of running top may break this weak affinity and cause more processes to  change  CPUs  more  often (because of the extra demand for cpu time).

%CPU  --  CPU usage    The  task's share of the elapsed CPU time since the last screen update, expressed as a percent-age of total CPU time.  In a true SMP environment, if  Irix mode is Off, top will operate in Solaris mode where a task's cpu usage will be divided by the total number of CPUs.

深入一些看PID-stat

Man of proc(5):

/proc/[pid]/stat
Status information about the process. This is used by ps(1). It is defined in the kernel source file fs/proc/array.c.

The fields, in order, with their proper scanf(3) format specifiers, are listed below.  Whether or not certain of these fields display valid information is governed by a ptrace access mode PTRACE_MODE_READ_FSCREDS | PTRACE_MODE_NOAUDIT check (refer to ptrace(2)).  If the check denies access, then the field value is displayed as 0. The affected fields are indicated with the marking [PT].

(1) pid  %d
The process ID.

(2) comm  %s
The filename of the executable, in parentheses. This is visible whether or not the executable is swapped out.

(3) state  %c
One of the following characters, indicating process state:
R  Running
S  Sleeping in an interruptible wait
D  Waiting in uninterruptible disk sleep
Z  Zombie
T  Stopped (on a signal) or (before Linux 2.6.33) trace stopped
t  Tracing stop (Linux 2.6.33 onward)
W  Paging (only before Linux 2.6.0)
X  Dead (from Linux 2.6.0 onward)
x  Dead (Linux 2.6.33 to 3.13 only)
K  Wakekill (Linux 2.6.33 to 3.13 only)
W  Waking (Linux 2.6.33 to 3.13 only)
P  Parked (Linux 3.9 to 3.13 only)

(4) ppid  %d
The PID of the parent of this process.

(5) pgrp  %d
The process group ID of the process.

(6) session  %d
The session ID of the process.

(7) tty_nr  %d
The controlling terminal of the process. (The minor device number is contained in the combination of bits 31 to 20 and 7 to 0; the major device number is in bits 15 to 8.)

(8) tpgid  %d
The ID of the foreground process group of the controlling terminal of the process.

(9) flags  %u
The kernel flags word of the process.  For bit meanings, see the PF_* defines in the Linux kernel source file include/linux/sched.h.  Details depend on the kernel version.

The format for this field was %lu before Linux 2.6.

(10) minflt  %lu
The number of minor faults the process has made which have not required loading a memory page from disk.

(11) cminflt  %lu
The number of minor faults that the process's waited-for children have made.

(12) majflt  %lu
The number of major faults the process has made which have required loading a memory page from disk.

(13) cmajflt  %lu
The number of major faults that the process's waited-for children have made.

(14) utime  %lu
Amount of time that this process has been scheduled in user mode, measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).  This includes guest time, guest_time (time spent running a virtual CPU, see below), so that applications that are not aware of the guest time field do not lose that time from their calculations.

(15) stime  %lu
Amount of time that this process has been scheduled in kernel mode, measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).

(16) cutime  %ld
Amount of time that this process's waited-for children have been scheduled in user mode, measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).  (See also times(2).)  This includes guest time, cguest_time (time spent running a virtual CPU, see below).

(17) cstime  %ld
Amount of time that this process's waited-for children have been scheduled in kernel mode, measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).

(18) priority  %ld
(Explanation for Linux 2.6) For processes running a real-time scheduling policy (policy below; see sched_setscheduler(2)), this is the negated scheduling priority, minus one; that is, a number in the range -2 to -100, corresponding to real-time priorities 1 to 99.  For processes running under a non-real-time scheduling policy, this is the raw nice value (setpriority(2)) as represented in the kernel. The kernel stores nice values as numbers in the range 0 (high) to 39 (low), corresponding to the user-visible nice range of -20 to 19.

Before Linux 2.6, this was a scaled value based on the scheduler weighting given to this process.

(19) nice  %ld
The nice value (see setpriority(2)), a value in the range 19 (low priority) to -20 (high priority).

(20) num_threads  %ld
Number of threads in this process (since Linux 2.6). Before kernel 2.6, this field was hard coded to 0 as a placeholder for an earlier removed field.

(21) itrealvalue  %ld
The time in jiffies before the next SIGALRM is sent to the process due to an interval timer.  Since kernel 2.6.17, this field is no longer maintained, and is hard coded as 0.

(22) starttime  %llu
The time the process started after system boot.  In kernels before Linux 2.6, this value was expressed in jiffies.  Since Linux 2.6, the value is expressed in clock ticks (divide by sysconf(_SC_CLK_TCK)).

The format for this field was %lu before Linux 2.6.

(23) vsize  %lu
Virtual memory size in bytes.

(24) rss  %ld
Resident Set Size: number of pages the process has in real memory.  This is just the pages which count toward text, data, or stack space.  This does not include pages which have not been demand-loaded in, or which are swapped out.

(25) rsslim  %lu
Current soft limit in bytes on the rss of the process; see the description of RLIMIT_RSS in getrlimit(2).

(26) startcode  %lu  [PT]
The address above which program text can run.

(27) endcode  %lu  [PT]
The address below which program text can run.

(28) startstack  %lu  [PT]
The address of the start (i.e., bottom) of the stack.

(29) kstkesp  %lu  [PT]
The current value of ESP (stack pointer), as found in the kernel stack page for the process.

(30) kstkeip  %lu  [PT]
The current EIP (instruction pointer).

(31) signal  %lu
The bitmap of pending signals, displayed as a decimal number.  Obsolete, because it does not provide information on real-time signals; use /proc/[pid]/status instead.

(32) blocked  %lu
The bitmap of blocked signals, displayed as a decimal number.  Obsolete, because it does not provide information on real-time signals; use /proc/[pid]/status instead.

(33) sigignore  %lu
The bitmap of ignored signals, displayed as a decimal number.  Obsolete, because it does not provide information on real-time signals; use /proc/[pid]/status instead.

(34) sigcatch  %lu
The bitmap of caught signals, displayed as a decimal number.  Obsolete, because it does not provide information on real-time signals; use /proc/[pid]/status instead.

(35) wchan  %lu  [PT]
This is the "channel" in which the process is waiting.  It is the address of a location in the kernel where the process is sleeping.  The corresponding symbolic name can be found in /proc/[pid]/wchan.

(36) nswap  %lu
Number of pages swapped (not maintained).

(37) cnswap  %lu
Cumulative nswap for child processes (not maintained).

(38) exit_signal  %d  (since Linux 2.1.22)
Signal to be sent to parent when we die.

(39) processor  %d  (since Linux 2.2.8)
CPU number last executed on.

(40) rt_priority  %u  (since Linux 2.5.19)
Real-time scheduling priority, a number in the range 1 to 99 for processes scheduled under a real-time policy, or 0, for non-real-time processes (see sched_setscheduler(2)).

(41) policy  %u  (since Linux 2.5.19)
Scheduling policy (see sched_setscheduler(2)). Decode using the SCHED_* constants in linux/sched.h.

The format for this field was %lu before Linux 2.6.22.

(42) delayacct_blkio_ticks  %llu  (since Linux 2.6.18)
Aggregated block I/O delays, measured in clock ticks(centiseconds).

(43) guest_time  %lu  (since Linux 2.6.24)
Guest time of the process (time spent running a virtual CPU for a guest operating system), measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).

(44) cguest_time  %ld  (since Linux 2.6.24)
Guest time of the process's children, measured in clock ticks (divide by sysconf(_SC_CLK_TCK)).

(45) start_data  %lu  (since Linux 3.3)  [PT]
Address above which program initialized and uninitialized (BSS) data are placed.

(46) end_data  %lu  (since Linux 3.3)  [PT]
Address below which program initialized and uninitialized (BSS) data are placed.

(47) start_brk  %lu  (since Linux 3.3)  [PT]
Address above which program heap can be expanded with brk(2).

(48) arg_start  %lu  (since Linux 3.5)  [PT]
Address above which program command-line arguments (argv) are placed.

(49) arg_end  %lu  (since Linux 3.5)  [PT]
Address below program command-line arguments (argv) are placed.

(50) env_start  %lu  (since Linux 3.5)  [PT]
Address above which program environment is placed.

(51) env_end  %lu  (since Linux 3.5)  [PT]
Address below which program environment is placed.

(52) exit_code  %d  (since Linux 3.5)  [PT]
The thread's exit status in the form reported by waitpid(2).

从C语言的角度可以这样看(procstat)

int           pid;                      /** The process id. **/
char          exName [_POSIX_PATH_MAX]; /** The filename of the executable **/
char          state; /** 1 **/          /** R is running, S is sleeping, D is sleeping in an uninterruptible wait, Z is zombie, T is traced or stopped **/
unsigned      euid,                      /** effective user id **/
egid;                      /** effective group id */                         
int           ppid;                     /** The pid of the parent. **/
int           pgrp;                     /** The pgrp of the process. **/
int           session;                  /** The session id of the process. **/
int           tty;                      /** The tty the process uses **/
int           tpgid;                    /** (too long) **/
unsigned int  flags;                    /** The flags of the process. **/
unsigned int  minflt;                   /** The number of minor faults **/
unsigned int  cminflt;                  /** The number of minor faults with childs **/
unsigned int  majflt;                   /** The number of major faults **/
unsigned int  cmajflt;                  /** The number of major faults with childs **/
int           utime;                    /** user mode jiffies **/
int           stime;                    /** kernel mode jiffies **/
int       cutime;                   /** user mode jiffies with childs **/
int           cstime;                   /** kernel mode jiffies with childs **/
int           counter;                  /** process's next timeslice **/
int           priority;                 /** the standard nice value, plus fifteen **/
unsigned int  timeout;                  /** The time in jiffies of the next timeout **/
unsigned int  itrealvalue;              /** The time before the next SIGALRM is sent to the process **/
int           starttime; /** 20 **/     /** Time the process started after system boot **/
unsigned int  vsize;                    /** Virtual memory size **/
unsigned int  rss;                      /** Resident Set Size **/
unsigned int  rlim;                     /** Current limit in bytes on the rss **/
unsigned int  startcode;                /** The address above which program text can run **/
unsigned int  endcode;                  /** The address below which program text can run **/
unsigned int  startstack;               /** The address of the start of the stack **/
unsigned int  kstkesp;                  /** The current value of ESP **/
unsigned int  kstkeip;                 /** The current value of EIP **/
int       signal;                   /** The bitmap of pending signals **/
int           blocked; /** 30 **/       /** The bitmap of blocked signals **/
int           sigignore;                /** The bitmap of ignored signals **/
int           sigcatch;                 /** The bitmap of catched signals **/
unsigned int  wchan;  /** 33 **/        /** (too long) **/
int       sched,        /** scheduler **/
sched_priority;       /** scheduler priority **/

进程状态在此可以看到:
List of states you can also have look into kernel source code fs/proc/array.c:

static const char * const task_state_array[] = {
/* states in TASK_REPORT: */
"R (running)",      /* 0x00 */
"S (sleeping)",     /* 0x01 */
"D (disk sleep)",   /* 0x02 */
"T (stopped)",      /* 0x04 */
"t (tracing stop)", /* 0x08 */
"X (dead)",     /* 0x10 */
"Z (zombie)",       /* 0x20 */
"P (parked)",       /* 0x40 */
/* states beyond TASK_REPORT: */
"I (idle)",     /* 0x80 */
};