Getting Meaningful Stopwatch Measurements

Since Stopwatch readings are derived from the internal system clock, they can be no more accurate than the clock itself. System clocks typically have a resolution of 1/60 second; Stopwatches based on such clocks yield measurements accurate to the nearest 1/60 second (about 17 ms). If the computation of interest is shorter than 1/60 second, the measurement may be off by as much as 100%. Programmers must often take steps to ensure their measurements are meaningful.

When using a Stopwatch for user, system, or real time measurements, the programmer must have three ``ballpark'' estimates in hand:

1. The duration of the computation of interest

   The programmer must know approximately how long the computation of interest takes in the units of time being measured (that is, user, system, or real). An estimate can always be obtained by making an initial measurement; if the measurement is non-zero, it can serve as the estimate; otherwise, the technique of repetition (described below) can be used to bootstrap an estimate.

2. The Stopwatch resolution

   The Stopwatch resolution determines the accuracy of the Stopwatch itself. The static member function Stopwatch::resolution() returns this (system-dependent) value in floating point seconds:

          #include <Stopwatch.h>
          #include <stream.h>
          main(){
              cout << Stopwatch::resolution() << "\ n";
          }
   

   which prints .016667 on my system (1/60 second, or about 17 ms, a typical value).

3. The timing bias inherent in the measurement itself

   When measuring user time, the measurement overhead will never exceed the time for a function call. System calls (if any) are counted toward system time.

   When measuring system time, however, the measurement overhead includes a system call. While a function call may take only a few microseconds, a system call may actually take several milliseconds; the exact value depends on factors which are usually impossible to predict unless the program being monitored is the only process in the system.

The key to getting meaningful Stopwatch measurement is to make sure the ratios of (1) to (2) and (1) to (3) are sufficiently large. If both ratios are large, say 100:1, no special steps need be taken. A ratio of 100:1 means that the reading is accurate to within 1%, more than adequate for typical performance-monitoring applications. Otherwise, the programmer must increase the ratios by manipulating the only variable under his or her control: the duration of the computation of interest. This can be done by measuring an adequate number of repetitions of the computation and then taking an average:

       const int N=10000;
       Stopwatch w;
       w.start();
       for(int i=0;i<N;i++){
           computation of interest    }
       w.stop(); // we have now measured 10,000 repetitions
       cout << "System time=" << w.system()/N << "\ n";