rtpm(1M)

rtpm -- real time performance monitor

Synopsis

rtpm [-h history_buffer_size] [interval]

Description

The command rtpm displays operating system performance metrics and usage information in an interactive real-time curses-based graphical display.

The interval argument specifies the time in seconds between successive samples of the performance information. The default interval is two seconds.

The history_buffer_size argument specifies the number of history data points rtpm saves for plotting metrics. The default value for the history buffer size is the number of columns on the screen, as specified by the $COLUMNS environment variable.

Screen size

Using the IOCGWINSZ ioctl system call, rtpm attempts to automatically determine your screen size and adjust the display accordingly. However, if the $LINES and $COLUMNS environment variables are set, they take precedence over the ioctl call. In no case will the screen size be smaller than 24 rows by 80 columns.

Screen layout

The rtpm screen is divided into three areas. The top portion of the screen shows the graphical display, the bottom line of the screen displays the status of the monitor, and 14 lines between the graphical display area and the status line display text-based metric information.

The graphical display area contains a bar graph of CPU consumption, plots of metric data, or both. The text-based metric display area consists of numeric metric data, labels for the metric data, and subscreen headers. The numeric metric data, subscreen headers, and plot titles are cursor addressable.

Views

The metrics are available in either of two views: a CPU view and a CG view. In the CPU view, the metrics are displayed on a per-processor basis. In the CG view, the metrics are displayed on a per-CPU group basis, where each CPU group consists of a set of processors.

You can toggle between views using the <g> or <G> key. The default view is the CPU view.

Status line

The bottom line of the screen is the status display area. Typically, it contains the name of the machine, as returned by the uname(2) system call, the current date and time, the requested sampling interval, and the actual interval sampled. Two other messages may be displayed on the status line. The first is the message LOCKED that means rtpm has been locked in memory by the plock(2) system call; only privileged users are allowed to do this. The second message is Enter <?> for help.

Cursor motions

Move the cursor by pressing any of the following case-insensitive keys:

the terminal arrow keys
<H>, <J>, <K>, <L> (as in vi)
<Ctrl><F>, <Ctrl>, <Ctrl>, <Ctrl><N> (as in emacs)

Plotting metrics

When the cursor is on a numeric metric data item, pressing either <Space> or the <Return> key causes the metric to be plotted in the graphical display area. The number of concurrent plots allowed depends on the size of the screen. A minimum of 7 rows and 40 columns in the graphical display area is required for a single plot. On a 24 row by 80 column screen, two concurrent plots can be displayed. Larger screens accommodate more plots.

When the cursor is placed on a plot (or bar graph) title, pressing either <Space> or the <Return> key removes the plot (or bargraph) from the graphical display area.

Pressing <C> clears whatever is in the upper-left corner of the graphical display area.

Scales for plots are determined by the maximum value contained in the history buffer at the time the plot is requested. If a later metric value is larger than the initial scale, the plot automatically re-scale to accommodate the larger value. Plots do not automatically re-scale to smaller values. Removing a plot and re-displaying it causes a new scale to be determined as above.

Displaying the bargraph

Use to toggle on and off the display of the CPU consumption bar graph.

Changing the sampling interval

The sampling interval can be set by an argument at invocation. The default sampling interval is two seconds. The sampling interval may be changed at any time by pressing either <+> or <->, which increments or decrements the interval by one second. The minimum sampling interval is one second. Users should note that a one second sampling interval places a moderate load on the system, and is not particularly useful for identifying system problems.

Getting help

Press <?> to display the help screen in the text portion of the screen. Press <Esc> to return to the previous screen.

Quitting rtpm

Press <Q> or <Ctrl><D> to exit rtpm.

Accessing subscreens

Press <Space> or <Return> while the cursor is on a subscreen header to change the text-based metric display to the subscreen. Press <Esc> to return to the previous screen. Twelve subscreens are available from the top level screen:

CPU: Per-processor/per-CG CPU consumption statistics.
CALLS: Per-processor/per CG system calls statistics.
MEMORY: Memory consumption and kernel memory allocator statistics.
PAGING: Paging and swapping statistics.
FILESYS: File system calls and tables, buffer cache, inode and directory block statistics.
IO: Per-disk I/O statistics.
TTY: Per-processor/per-CG terminal I/O statistics.
QUEUE: Run and swap queue statistics, and per-processor local run queue and process switching statistics. The CG view is not supported for these statistics.
LWPS: Process and LWP (light weight process) statistics.
ETHER: Device level ethernet networking statistics.
TCP/IP: TCP/IP networking statistics.

Five of the twelve screens accessible from the top level screen contain further subscreens that can be displayed:

PAGING

PAGE IN: Per-processor/per-CG page in statistics
PAGE OUT/SWAPPING: Per-processor/per CG page out and swapping statistics

FILESYS

FILE SYS CALLS: Per-processor/per CG filesystem calls and directory name lookup cache statistics
BUFFER CACHE: Per-processor/per CG buffer cache statistics
MISC/TABLES: Per-filesystem inode table statistics and file table, file lock table, and wio statistics
IGETS/DIRBLKS: Inode get and directory block statistics that are reported per processor/perCG and filesystem type.
INODE RECLAIMS: Inode reclaims with pages and without pages that are reported per processor/per CG and filesystem type.

ETHER

ETHERNET: Per-device ethernet packet and octet rates and queue lengths.
INPUT ERRORS: Per-device ethernet input errors.
OUTPUT ERRORS: Per-device ethernet output errors.

TCP/IP

ICMP: ICMP statistics.
TCP: TCP statistics.
IP: IP statistics.

Display example

The default startup display looks like this:

 CPU 0  ====================================================-------------------
 CPU 1  ===-------
 CPU 2  -
 CPU 3  -
 total  ============================------------
%s= _%u-    |______________10|_ _ _ _ _2_0_|_ _ _ _ _3_0_|_ _ _ _ _4_0_|_ _ _ _ _5_0_|_ _ _ _ _6_0_|_ _ _ _ _7_0_|_ _ _ _ _8_0_|_ _ _ _ _9_0_|_ _ _ _1_0_0_|_

      CPU:       CALLS/s:      IO/s:        QUEUE:       TTY/s:       ETHER:
     58 %cpu    3460 calls      0 reads      1 runq       0 rcvs       1 xpkt/s
     42 %usr       0 forks      0 rdblk    100 %run       0 xmit       1 rpkt/s
     18 %sys       0 execs      0 writs      0 prunq      0 mdms     211 xoct/s
     40 %int      20 reads      0 wrblk      0 %prun      0 canch     60 roct/s
      0 %wio       2 writs      0 qlen       0 swpq       0 rawch      0 xerrs
     42 %idl       0 Krwch      0 %busy      0 %swp     143 outch    496 rerrs

    MEMORY:     PAGING/s:   FILESYS/s:      LWPS:       NETWARE:     TCP/IP:
    973 kma        0 pgins      0 igets     54 lwps       0 spx/s      2 tcp/s
  10161 frmem      0 pgots    684 lkups      0 run        0 ipx/s      0 udp/s
  32149 frswp      0 atchs      0 dirbk     52 sleep      0 sap/s      0 icmp/s
     38 %mem       0 pflts    100 %dnlc      0 zomb       0 rip/s      2 ip/s
      8 %swp       0 vflts    161 inode     30 procs      0 errs       1 errs

UNIX_SV lycia            Thu Sept 1 16:04:11 1998     interval:    1 (1.00)

In the default startup display, a bar graph of CPU utilization is displayed in the graphical portion of the screen. When rtpm is run on a multi-processor system, there is a bargraph for each CPU on the system and one for total CPU consumption. Systems with more than four processors require a screen size that contains more than 24 rows to display the entire bar graph. When rtpm is run on a small screen and is displaying statistics for a large multi-processor configuration, not all CPUs are displayed in the bar graph. A scroll bar at the left of the bar graph shows whether the bar graph can be scrolled up or down. The bar graph may be scrolled up and down with the <^> and <V> keys. In the CG view, the bar graph is displayed for CG-based CPU usage.

Pressing the key when the CPU bargraph is displayed reports the percentage time spent processing interrupts; pressing again turns off the display of interrupt data. Time spent in interrupt handling is represented by the <+> character as shown in the following display.

 CPU 0  ====================================================++++++-------------
 CPU 1  ===-------
 CPU 2  -
 CPU 3  -
 total  ============================+++---------
s= i+ u-   |______________10|_ _ _ _ _2_0_|_ _ _ _ _3_0_|_ _ _ _ _4_0_|_ _ _ _ _5_0_|_ _ _ _ _6_0_|_ _ _ _ _7_0_|_ _ _ _ _8_0_|_ _ _ _ _9_0_|_ _ _ _1_0_0_|_

If the bargraph is cleared and the percentage cpu (usr+sys+int) time metric is plotted, the display looks like this:

%cpu total
100
 80                                                ||||  ||            ||||
 60 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 40 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 20 |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

      CPU:       CALLS/s:      IO/s:        QUEUE:       TTY/s:       ETHER:
     58 %cpu    3478 calls      0 reads      1 runq       0 rcvs       1 xpkt/s
     42 %usr       0 forks      0 rdblk    100 %run       0 xmit       1 rpkt/s
     15 %sys       0 execs      0 writs      0 prunq      0 mdms     291 xoct/s
     43 %int      20 reads      0 wrblk      0 %prun      0 canch    154 roct/s
      0 %wio       2 writs      0 qlen       0 swpq       0 rawch      0 xerrs
     42 %idl       0 Krwch      0 %busy      0 %swp     223 outch    499 rerrs

    MEMORY:     PAGING/s:   FILESYS/s:      LWPS:       NETWARE:     TCP/IP:
    974 kma        0 pgins      0 igets     56 lwps       0 spx/s      2 tcp/s
  10814 frmem      0 pgots    687 lkups      0 run        0 ipx/s      0 udp/s
  32113 frswp      0 atchs      0 dirbk     54 sleep      0 sap/s      0 icmp/s
     34 %mem       0 pflts    100 %dnlc      0 zomb       0 rip/s      2 ip/s
      8 %swp       0 vflts    162 inode     32 procs      0 errs       1 errs

UNIX_SV lycia            Thu Sept 1 16:10:10 1998     interval:    1 (1.00)

Several plotting types are available by entering the key; the above screen shows a vertical bar plot, and the following is an example of a scatter plot that consists of a single plotted point per sample.

%cpu total
100                                                                     *
 80 ******************************************************************** ******
 60
 40
 20
  0 ___________________________________________________________________________

Accessing the LWP subscreen displays statistics about the number of processes and light weight processes, as well as some ps-like information about the LWPs that have run during the sampling interval. In this example, a runaway user process is consuming over half of the available CPU cycles:

% cpu total 100 80 ** 60 ************************************************ ************************* 40 20 0 ___________________________________________________________________________

56 lwps 0 runnable lwps 0 zombie lwps 32 procs 2 lwps on CPU 54 sleeping lwps 0 idle lwps 400 procmax 0 lwpfail 0 stopped lwps 0 other lwps 0 procfail

%%% S USER PID LWPID CPU PRI CPUTIME SIZE TTY CMD[LWP] 51 O root 375 1 0 0 433:47.00 4339 ? runaway_proc 7 O root 728 1 1 53 2:52.24 4592 pts/3 ./rtpm 1 0 S root 0 23 1 79 0:25.77 0 ? sysproc[vx_inact_da 0 S root 0 19 1 79 0:00.23 0 ? sysproc[vx_inact_da 0 S root 0 2 1 79 4:02.75 0 ? sysproc[fsflushwp] 0 S root 0 14 1 79 2:12.47 0 ? sysproc[vx_flush_da

UNIX_SV lycia Thu Sept 1 16:28:18 1998 interval: 1 (1.00)

By default, the LWP subscreen displays all processes that have been active during the sampling period. Pressing or <S> while the LWP subscreen is displayed limits the data to user or system processes, respectively. Pressing the <A> key reports all active user and system processes. The <e> key acts as a toggle switch: pressing <E> once displays all processes, including sleeping processes; pressing <E> again causes only active processes to be displayed. The keys can be used in combinations: pressing <A> and <E> will show all of the processes (including sleeping processes) that are on the system, pressing and <E> will show all of the user processes, and pressing <S> and <E> will show all of the system processes. If more processes run during the sampling interval than will fit on the screen, the <<> and <>> keys can be used to scroll through the list.

In the CG view, an extra column is added to the PS output to display the CG on which a lwp is executing.

On systems with a large number of CPUs, disks, or ethernet cards, the per-unit metrics associated with these resources may not fit on a small screen. When this happens, <-, ->, or <> will be displayed at the right edge of the screen and the <<> and <>> keys can be used to scroll the per-resource metrics left and right. Moving the cursor off the edge of the screen also scrolls per-resource metrics.

Command summary

Cursor Motions: The arrow keys, <H>, <J>, <K>, <L>, and <Ctrl>, <Ctrl><N>, <Ctrl>, <Ctrl><F> move the cursor left, down, up, and right, respectively.
Changing Views: You can toggle between the CPU and CG views using the <g> or <G> key. The default view is the CPU view.
Plotting Metrics: Press <Space> or the <Return> key while the cursor is on a numeric metric to plot the metric. Enter to change the plotting format. Press the <Space> bar or the <Return> key while the cursor is on a plot title to delete the plot. Pressing <C> deletes the plot (or bar graph) in the upper left corner of the screen.
Accessing Subscreens: Press <Space> or the <Return> key while the cursor is on a subscreen header to display the subscreen associated with the header. Press <Esc> to return to the previous screen.
Help: Enter <?> to display the help screen. Press <Esc> to return to the previous screen.
Locking rtpm into Memory: Privileged users can use <X> as a toggle to lock or unlock rtpm in memory.
Bargraph: Press to toggle on and off the display of the percentage CPU consumption bar graph. On multi-processing systems that cannot fit information about all their processors/CGs within the available display area, the <^> and <V> keys scroll the bar graph up and down.
Changing the Sampling Interval: The <+> and <-> keys increment and decrement the sampling interval.
Scrolling Metrics: When the text display area is not large enough to display all per-resource metrics, scroll the display using the cursor keys or the <<> and <>> keys.
LWP Screen: Scroll the LWP screen that displays the ps data using the <<> and <>> keys. The ps report can be limited to user or system, or to show all processes by the , <S>, and <A> keys, respectively.
Underscore Handling: Some termcap entries do not handle the terminal underscore capability correctly. Use the <_> key to toggle between selecting whether rtpm tries to use the underscore attribute or draws an ASCII underscore character.
Redraw: Enter <Ctrl><L> to redraw the screen.
Exiting rtpm: Enter <Q> or <Ctrl><D> to exit rtpm.

Initialization file

When rtpm is invoked, it attempts to read the .rtpmrc initialization file from the user's home directory. If the .rtpmrc file is not found in the user's home directory, the system default /etc/.rtpmrc is used.

The native language in which the .rtpmrc file is written is specified by the expression LANG=language, where language is a locale name describing the message catalog to be used for reading the initialization file. For example, setting LANG=C specifies the default English language strings as used in /etc/.rtpmrc. Note that setting LANG in the .rtpmrc file does not affect the messages displayed by rtpm.

The initialization file can be used to specify a default set of metrics to plot at initialization time, and also specifies the display colors for color terminals. Each line of the initialization file consists of a name, a colon, and an expression that specifies color or default plotting status.

The following display elements are specified in the initialization file:

background: the background color.
default: the default numeric metric display color.
plot: the default plotting color.
labels: the display color for metric and plot labels.
headers: the display color for cursor addressable subscreen headers and plot titles.
messages: the display color for informational and error messages.
bargraph: the display color range for the %CPU bar graph.
metric names: the display color ranges and plotting status for individual metrics.

Each of the above display elements may be assigned one of the following colors: magenta, red, yellow, green, cyan, blue, black, and white. The bargraph and metrics can be assigned a color range expression, and metrics can be assigned a plotting status.

Color range expressions are used to assign display colors to ranges of metric values. A color range expression consists of a series of numbers, relational operators, and colors:

   <color_range>::
   <RANGE>[;<RANGE>]
   <RANGE>::
   	[<num><OP>]<color>[<OP><num><OP><color>]*[<OP><num>]
   <OP>::
   	< | <= | > | >= | == | !=

For example

   bargraph: 0 <= green < 60 <= yellow < 85 <= red

sets the bargraph to green if CPU consumption is below 60%, yellow if it is between 60% and 85%, and red if it is greater than 85%.

Multiple expressions may be used, provided they are separated by semicolons. Since expressions are tested from left to right, the following expression is equivalent to the one above:

   bargraph: green < 60 ; yellow < 85 ; red

In the text-based metric display area, the color assigned to default is used to display the numeric metric unless a color expression specifies otherwise. In the plot area, the color assigned to plot is used to display the plot characters unless a color expression specifies otherwise. In both the text-based and graphical display areas, if a color expression is true, the metric will be displayed in the color defined by the expression. Typically, colors will vary from green to yellow to red depending on the severity of a condition. On vertical bar chart plots, this will have the effect of several color bands, one above the other.

For metrics that indicate an alarm condition when they approach 0, such as free memory, the user may want to invert the plot so that 0 is displayed as the top of the y-axis as opposed to the bottom. Thus the plot will only display the alarm color under severe conditions.

To invert a plot, add the invert keyword to a metric in the initialization file:

   freemem: 1000 >= yellow > 500 >= red ; invert

Plot status expressions are used to specify metrics that are to be plotted when rtpm is invoked. Since metrics can be per-resource based, a metric is actually a set of values called instances. A metric has an instance for each resource (or combination of resources) on which it is based. For example, freemem is global, %usr time is kept per-CPU, and igets/s are kept both per-CPU and per-filesystem. Hence, there is one instance for freemem, n CPU instances for %usr, and n CPU * nfstyp instances for igets/s. A plot expression is used to specify the set of instances the user wants to plot. Here are some examples of plot expressions:

plot: plot a single global metric or the total of a resource based metric.
plot(total): plot the total of a metric that has one resource.
plot(number): plot the instance associated with resource number such as plot(2)) to plot the value associated with CPU #2.
plot(*): plot all the instances associated with this single dimensional metric.
plot(2, 3): plot the instance associated with the first resource of 2 and the second resource of 3, such as igets/s for CPU #2 and filesys #3.

Multiple plot expressions are separated by semi-colons. If there is not enough room on the screen for the requested plots, the ones specified last in the initialization file are displayed, and no error message is generated.

Usage note

On heavily loaded systems where memory is scarce, and especially with large screen sizes, rtpm may fail to allocate enough space to save all its history points. If this is the case, the history buffer size should be decreased using the -h option. On systems with ample memory, large display screens may benefit from increasing the history buffer size to the number of columns on the screen.

List of metrics

The following is a list of all the metrics displayed by rtpm:

Machine resource information

CPU: the number of processors on the system
disk: the number of disk drives on the system
fstype: the number of filesystem types in the system
fsnames: the names of the filesystems in the system
kmpool: the number of kernel memory allocator pools in the system
kmasize: the sizes of the kernel memory allocator pools
pgsz: the page size of the system
dsname: the names of the disk drives on the system
nether: the number of ethernet cards on the system
ethname: the names of the ethernet devices on the system

CPU consumption statistics

%cpu: the percentage of user, system time and time spent processing interrupts (per CPU/per CG)
%(wio+idl): the waiting for I/O and idle time (per CPU/per CG)
%usr: the percentage of user time (per CPU/per CG)
%sys: the percentage of system time (per CPU/per CG)
%int: the percentage of system time spent processing interrupts (per CPU/per CG)
%wio: the percentage of waiting for I/O time (per CPU/per CG)
%idl: the percentage of idle time (per CPU/per CG)

Filesystem statistics

iget/s: the number of inode get operations per second (per CPU/per CG and filesystem)
dirblk/s: the number of directory blocks read per second (per CPU/per CG and filesystem)
ipage/s: the number of inodes reclaimed with associated pages per second (per CPU/per CG and filesystem)
inopage/s: the number of inodes reclaimed without associated pages per second (per CPU/per CG and filesystem)
fswio: the current number of outstanding filesystem i/o jobs
physwio: the current number of outstanding physical i/o jobs
fltblinuse: the number of file table entries in use
fltblfail: the number of failed attempts to get a file table entry
fltblfail/s: the number of failed attempts to get a file table entry per second
flcktblmax: the maximum number of file lock table entries
flcktbluse: the number of file lock table entries in use
flcktblfal: the number of failed attempts to get a file lock table entry
flcktblfail/s: the number of failed attempts to get a file lock table entry per second
flcktbl/s: the number of attempts to get a file lock table entry per second
maxinode: the maximum number of inodes allowed (per filesystem)
currinode: the current number of inodes allocated (per filesystem)
inodeinuse: the number of inodes in use (per filesystem)
inodefail: the number of failed attempts to allocate an inode (per filesystem)

Memory statistics

freemem: the amount of free memory in the system in pages
freeswp: the amount of free swap memory in the system in pages
mem: the amount of memory used by the kma pools (per pool)
balloc: the amount of memory allocated by the kma pools (per pool)
ralloc: the amount of memory requested of the kma pools (per pool)
kmfail: the number of failed kma requests (per pool)
kma(pg): the number of pages being used by the kernel memory allocated
%mem: the percentage of memory in use
%memswp: the percentage of memory swap space in use
%dskswp: the percentage of disk swap space in use
dskswp: the number of pages of disk swap space in the system
dskfreeswp: the number of pages of free disk swap space in the system
memswp: the number of swap memory pages in the system
totalmem: the number of memory pages in the system

Paging and swapping statistics

preatch/s: the number of pre-attaches per second (per CPU/per CG)
atch/s: the number of attaches per second (per CPU/per CG)
atchfree/s: the number of attaches from the free list per second (per CPU/per CG)
atfrpgot/s: the number of attaches from the free list which resulted in a page out operation per second (per CPU/per CG)
atchmiss/s: the number of attach misses (per CPU/per CG)
pgin/s: the number of page in operations per second (per CPU/per CG)
pgpgin/s: the number of pages paged in per second (per CPU/per CG)
pgout/s: the number of page in operations per second (per CPU/per CG)
pgpgout/s: the number of pages paged out per second (per CPU/per CG)
swpout/s: the number of swap out operations per second (per CPU/per CG)
ppgswpot/s: the number of physical pages swapped out per second (per CPU/per CG)
vpgswpot/s: the number of virtual pages swapped out per second (per CPU/per CG)
swpin/s: the number of swap in operations per second (per CPU/per CG)
pgswpin/s: the number of pages swapped in per second (per CPU/per CG)
virscan/s: the number of pages scanned by page freeing algorithms per second (per CPU/per CG)
virfree/s: the number of virtual pages freed by page freeing algorithms per second (per CPU/per CG)
physfree/s: the number of physical pages freed by page freeing algorithms per second (per CPU/per CG)
pfault/s: the number of protection faults per second (per CPU/per CG)
vfault/s: the number of validity faults per second (per CPU/per CG)
sftlck/s: the number of software lock operations per second (per CPU/per CG)

Process switching and queueing statistics

pswtch/s: the number of process switches per second (per CPU/per CG)
runq: the mean run queue length
%runocc: the percentage of time the run quqe was occupied
swpq: the mean swap queue length
%swpocc: the percentage of time the swap queue was occupied
prunq: the length of the processor local run queue (per CPU/per CG)
%prunocc: the percentage time the processor local run queue was occupied (per CPU/per CG)

System call statistics

syscall/s: the number of system calls per second (per CPU/per CG)
fork/s: the number of fork calls per second (per CPU/per CG)
lwpcreat/s: the number of lwpcreate calls per second (per CPU/per CG)
exec/s: the number of exec calls per second (per CPU/per CG)
read/s: the nubmer of read calls per second (per CPU/per CG)
write/s: the number of write calls per second (per CPU/per CG)
readch/s: the number of characters read per second (per CPU/per CG)
writech/s: the number of characters written per second (per CPU/per CG)
lookup/s: the number of filename lookup operations per second (per CPU/per CG)
dnlchits/s: the number of directory name lookup cached hits per second (per CPU/per CG)
dnlcmiss/s: the number of directory name lookup cache misses per second (per CPU/per CG)
bread/s: the number of blocks read into the buffer cache per second (per CPU/per CG)
bwrite/s: the number of blocks written from the buffer cache per second (per CPU/per CG)
lread/s: the number of logical blocks read from the buffer cache per second (per CPU/per CG)
lwrite/s: the number of logical blocks written to the buffer cache per second (per CPU/per CG)
phread/s: the number of physical read operations per second (per CPU/per CG)
phwrite/s: the number of physical write operations per second (per CPU/per CG)
ipcmsgq/s: the number of ipc message queues sent and received per second (per CPU/per CG)
ipcsema/s: the number of ipc semaphore operations per second (per CPU/per CG)
(rd+wrt)/s: the number of read and write system calls per second
(r+w)Kb/s: the number of characters read and characters written (in Kbytes) per second
%dnlc: the percentage of directory name lookup cache hits (per CPU/per CG)
%rcache: the percentage of buffer cache reads that were satisfied by a read from the buffer cache (per CPU/per CG)
%wcache: the percentage of buffer cache writes that wrote to a block in the buffer cache (per CPU/per CG)

Terminal I/O statistics

rcvint/s: the number of receiver interrupts per second (per CPU/per CG)
xmtint/s: the number of transmitter interrupts per second (per CPU/per CG)
mdmint/s: the number of modem interrupts per second (per CPU/per CG)
rawch/s: the number of raw characters written to ttys per second (per CPU/per CG)
canch/s: the number of canonical characters written to ttys per second (per CPU/per CG)
outch/s: the number of output characters to tty per second (per CPU/per CG)

Process and LWP (Light weight process) statistics

procfail: the number of failed attempts to get a process table entry
procinuse: the number of process table entries in use
procmax: the maximum number of process table entries
lwpfail: the number of failed attempts to create a lwp
lwpinuse: the number of lwps in uses
lwpmax: the maximum number of lwps allowed
lwp_sleep: the number of sleep lwps
lwp_run: the number of runnable lwps
lwp_idle: the number of idle lwps
lwp_onproc: the number of lwps currently on a processor/CG
lwp_zombie: the number of zombied lwps
lwp_stop: the number of stopped lwps
lwp_other: the number of other lwps
lwp_total: the total number of lwps
lwp_nproc: the number of processes in the system

Disk statistics

The following statistics are given for each disk. Pressing <Enter>, or the space bar, on a disk header gives the statistics for the individual slices of that disk.

instqlen: the instantaneous disk queue length (per disk)
%busy: the percentage time the disk was busy (per disk)
avgqlen: the mean disk queue length (per disk)
dsread/s: the number of disk read operations per second (per disk)
dsrblk/s: the number of disk blocks read per second (per disk)
dswrit/s: the number of disk write operations per second (per disk)
dswblk/s: the number of disk blocks written per second (per disk)

STREAMS statistics

streams: the number of streams allocated in the system
queues: the number of streams queues in the system
mdbblks: the number of streams message data blocks in the system
msgblks: the number of streams message blocks in the system
links: the number of streams links in the system
events: the number of streams events in the system
eventfail: the number of failed streams events

Ethernet statistics

InUcastPkts/s: the number of ethernet packets received per second (per ethernet device)
OutUcastPkts/s: the number of ethernet packets transmitted per second (per ethernet device)
InNUcastPkts/s: the number of ethernet broadcast packets received per second (per ethernet device)
OutNUcastPkts/s: the number of ethernet broadcast packets transmitted per second (per ethernet device)
InOctets/s: the number of ethernet octets received per second (per ethernet device)
OutOctets/s: the number of ethernet octets transmitted per second (per ethernet device)
InErrors: the number of ethernet input errors (per ethernet device)
etherAlignErrors: the number of ethernet frame alignment errors (per ethernet device)
etherCRCerrors: the number of ethernet checksum errors (per ethernet device)
etherOverrunErrors: the number of overrun errors (per ethernet device)
etherUnderrunErrors: the number of underrun errors (per ethernet device)
etherMissedPkts: the number of missed packet errors (per ethernet device)
InDiscards: the number of good packets discarded (per ethernet device)
etherReadqFull: the number of good packets discarded because the read queue was full (per ethernet device)
etherRcvResources: the number of good packets discarded because resources were not available (per ethernet device)
etherCollisions: the number of ethernet collision errors (per ethernet device)
OutDiscards: the number of ethernet output packets discarded (per ethernet device)
OutErrors: the number of ethernet output errors (per ethernet device)
etherAbortErrors: the number of ethernet abort errors (per ethernet device)
etherCarrierLost: the number of ethernet carrier lost errors (per ethernet device)
OutQlen: the ethernet output queue length (per ethernet device)

General networking statistics

ip_sum/s: the number of IP packets transmitted and received per second
icmp_sum/s: the number of ICMP messages transmitted and received per second
udp_sum/s: the number of UDP packets transmitted and received per second
tcp_sum/s: the number of TCP packets transmitted and received per second
neterr_sum: the sum of networking errors for IP, ICMP, UDP and TCP
neterr_sum/s: the number of networking errors per second

IP (Internet protocol) networking statistics

ip_total/s: the number of packets transmitted and received per second
ip_badsum: the number of packets received with a bad header check sum
ip_tooshort: the number of packets received that were too short
ip_toosmall: the number of packets received whose data size was too small
ip_badhlen: the number of packets received whose header length was wrong
ip_badlen: the number of packets received whose data length was wrong
ip_unknownproto: the number of packets received with an unknown protocol
ip_fragments: the number of fragments received
ip_fragdropped: the number of fragments dropped
ip_fragtimeout: the number of fragments timed out
ip_reasms: the number of packets re-assembled from fragments
ip_forward: the number of packets forwarded
ip_cantforward: the number of packets that could not be forwarded
ip_noroutes: the number of packets with no-routing information
ip_redirectsent: the number of packet redirects sent
ip_inerrors: the number of input errors
ip_indelivers/s: the number of packets delivered per second
ip_outrequests/s: the number of output requests per second
ip_outerrors: the number of output errors
ip_pfrags: the number of fragmented packets created
ip_frags: the number of fragments created
ip_fragfails: the number of failed attempts to fragment

Internet Control Message Protocol (ICMP networking statistics

icmp_intotal/s: the number of messages received per second
icmp_outtotal/s: the number of messages sent per second
icmp_reflect/s: the number of message responses sent per second
icmp_outerrors: the number of output system errors
icmp_error: the number of errors
icmp_oldicmp: the number of errors due to message of old ICMP type
icmp_badcode: the number of bad code field errors
icmp_tooshort: the number of errors due to message being too short
icmp_checksum: the number of check sum errors
icmp_badlen: the number of messages with bad length
icmp_echo_reply_in: the number of echo replies received
icmp_echo_reply_out: the number of echo replies sent
icmp_dest_unreachable_in: the number of input packets with unreachable destinations
icmp_dest_unreachable_out: the number of output packets with unreachable destinations
icmp_source_quench_in: the number of input source quenches
icmp_source_quench_out: the number of output source quenches
icmp_routing_redirects_in: the number of routing redirects received
icmp_routing_redirects_out: the number of routing redirects sent
icmp_echo_in: the number of echoes received
icmp_echo_out: the number of echoes sent
icmp_time_exceeded_in: the number of time outs received
icmp_time_exceeded_out: the number of time outs sent
icmp_parameter_problems_in: the number of parameter problems received
icmp_parameter_problems_out: the number of parameter problems sent
icmp_time_stamp_in: the number of time stamp requests received
icmp_time_stamp_out: the number of time stamp requests sent
icmp_time_stamp_reply_in: the number of time stamp replies received
icmp_time_stamp_reply_out: the number of time stamp replies sent
icmp_info_request_in: the number of information requests received
icmp_info_request_out: the number of information requests sent
icmp_info_reply_in: the number of information replies received
icmp_info_reply_out: the number of information replies sent
icmp_address_mask_request_in: the number of address mask requests received
icmp_address_mask_request_out: the number of address mask requests sent
icmp_address_mask_reply_in: the number of address mask replies received
icmp_address_mask_reply_out: the number of address mask replies sent

Transport Control Protocol (TCP) networking statistics

tcp_sndtotal/s: the number of packets sent per second
tcp_sndpack/s: the number of data packets sent per second
tcp_sndbyte/s: the number of bytes sent per second
tcp_sndrexmitpack: the number of data packets retransmitted
tcp_sndrexmitbyte: the number of data bytes retransmitted
tcp_sndacks: the number of ack only packets sent
tcp_delack: the number of delayed ack only packets sent
tcp_sndurg: the number of URG only packets sent
tcp_sndprobe: the number of window probe packets sent
tcp_sndwinup: the number of window update packets sent
tcp_sndctrl: the number of control packets sent
tcp_rcvtotal/s: the number of packets received per second
tcp_rcvackpack/s: the number of acks received per second
tcp_rcvackbyte/s: the number of bytes acked per second
tcp_rcvdupack: the number of duplicate acks received
tcp_rcvacktoomuch: the number acks received for unsent data
tcp_rcvpack: the number of in sequence packets received
tcp_rcvbyte: the number of in sequence bytes received
tcp_rcvduppack: the number of completely duplicate packets received
tcp_rcvdupbyte: the number of completely duplicate bytes received
tcp_rcvpartduppack: the number of partially duplicated packets received
tcp_rcvpartdupbyte: the number of partially duplicate bytes received
tcp_rcvoopack: the number of out of order packets received
tcp_rcvoobyte: the number of out of order bytes received
tcp_rcvpackafterwin: the number of packets received after window close
tcp_rcvbyteafterwin: the number of bytes received after window close
tcp_rcvwinprobe: the number of window probes received
tcp_rcvwinupd: the number of window updates received
tcp_rcvafterclose: the number of packets received after close
tcp_rcvbadsum: the number of packets discarded for bad check sum
tcp_rcvbadoff: the number of packets discarded for bad header offset fields
tcp_rcvshort: the number of packets discarded because packet was too short
tcp_connattempt: the number of connections requests
tcp_accepts: the number of accepted connections
tcp_connects: the number of connections established (including accepts)
tcp_closed: the number of connections closed
tcp_drops: the number of connections drops
tcp_conndrops: the number of embryonic connections dropped
tcp_rttupdated: the number of segments with updated round trip times
tcp_segstimed: the number of attempts to get round trip times
tcp_rexmttimeo: the number of retransmit timeouts
tcp_timeoutdrop: the number of connections dropped by retransmit timeout
tcp_persisttimeo: the number of persist timeouts
tcp_keeptimeo: the number of keepalive timeouts
tcp_keepprobe: the number of keepalive probes
tcp_keepdrops: the number of connections dropped by keepalive

User Datagram Protocol (UDP) networking statistics:

udp_hdrops: the number of incomplete headers
udp_badlen: the number of bad data length fields
udp_badsum: the number of check sums
udp_fullsock: the number of full sockets
udp_noports: the number of bad ports
udp_indelivers/s: the number of input packets delivered per second
udp_inerrors: the number of system errors during input
udp_outtotal/s: the number of output packets sent per second