Measuring Unix kernel performance

Measuring Unix Kernel PerformanceM.ShandAbstractWe present accurate,low-level measurements of process preemption,interrupt han-dling and memory system performance of a UNIX1workstation.To gather this data,we use PAMs(Programmable Active Memory).These are fast, general purpose,bit-level programmable co-processors based onfield programmable gatearrays.They are mapped to part of the system address space and appear to the CPU asmemory,much like memory-mapped I/O devices.PAMs are primarily aimed at computationally intensive problems,where wide, application specific datapaths can offer great speedups over software.By contrast,in thisapplication we rely on the realtime,concurrent aspects of a PAM that is relatively modestin terms of computational resources.Starting with a simple25MHz counter,we describe a series of measurement devices built to answer specific questions about low-level system performance.Many of thedevices are active in that they provoke the events they seek to measure.Our measurementtechniques allow us to construct histograms gradated in CPU clock cycles and cover:frequency and duration of user process preemption under various workloads,latency frominterrupt to kernel handler,nesting of interrupts,DMA throughput and latency and theeffect of other system activity on DMA.1IntroductionIn the design of high performance computer peripherals designers must pay attention not only to the basic capabilities of the host hardware,but also to characteristics of the operating system. Parameters such as the time to service a device interrupt can fundamentally affect the viability of competing designs.To some extent these parameters can be determined by analytical models or by simple expedients such as counting instructions in the perceived critical path, but ultimately the most reliable method is measurement.Unfortunately measuring activity at the lowest levels of computer systems can be extremely difficult.The phenomena under measure occur in far less time than the resolution of standard clock devices and involve system components like the cache that are largely hidden from the CPU.Recently we have seen much progress in instrumentation of programs and even entire systems[BKW90]by code modification[Wall91],but these methods can greatly perturb the objects being measured and the techniques are by no means easy to implement.The alternative Digital Equipment Corp.,Paris Research Laboratory,85Av.Victor Hugo.92500Rueil-Malmaison,France.1Unix is a registered Trademark of AT&T.1of adding purpose built measurment hardware[CBK88][EC84]tends to be either inflexible, expensive,or both,and is best adapted to systems using a microcoded implementation.For the purposes of characterizing the performance of a few parts of the kernel,neither of the above approaches seems justified in terms of the time and effort they involve.Even adding a high resolution clock is an excessive burden if new hardware must be built which willfind no further use once the measurements are made.1.1PAM TechnologyPAM technology,introduced by Vuillemin[BRV89],is based on a matrix of programmable active bits.It permits the realisation of synchronous logic circuits,each of the registers being updated on each cycle of a global clock signal.The maximum clock speed for such a design is directly determined by its critical combinatorial path,which varies from one design to another. When implemented thoughfield programmable gate arrays such as the Xilinx3000series [Xi89],it is not difficult to realize circuits that operate at25to30MHz.PAM stands for Programmable Active Memory.It is infinitely reprogrammable thanks to thefield programmable gate arrays from which it is built.The rest of the acronym derives from its role as a coprocessor occupying part of a host address space,receiving commands and returning results in response to the address and data transactions the host directs towards it.Being mapped directly into user and kernel address spaces,the CPU communicates very cheaply with the PAM making it suitable for the implementation of low-level measurement devices.Being reuseable and easy to program makes it suitable for the implementation of specialized single use devices.PAMs are primarily aimed at computationally intensive problems,where wide,appli-cation specific datapaths can offer great speedups over software[SBV89].By contrast, in measurement applications we rely on the realtime,concurrent aspects of a PAM that is relatively modest in terms of computational resources.In fact we have just such a PAM at our disposal—3mint,the interface module of our next generation computationally oriented PAM.In normal operation3mint hasfixed programming loaded out of PROM,but it too uses a Xilinx gate array,and by providing the ability to alter its programming we obtain a small PAM connected directly to the system I/O channel.1.2TURBOchannelThe particular I/O channel in question is TURBOchannel[TbC91].TURBOchannel is a synchronous,asymetric I/O channel operating at12.5to25MHz.It connects one system module containing CPU and memory to some number of option modules.TURBOchannel supports two kinds of transaction:an I/O transaction in which the system module can read or write an option module,and a DMA transaction in which an option module can read or write the system module.I/O transactions are relatively straightforward;they are issued in response to CPU memory transactions to that region of the address space to which an option is mapped.Writes are allowed to proceed asynchronously and may be issued in one cycle;sustained throughput is one word every three cycles,to minimum latency options.Reads must stall the processor until the option responds.On the DECstation5000/200implementation of TURBOchannel this isa minimum of8cycles.This asymetry means that scattered I/O writes place a much lighter burden on the CPU than I/O reads.DMA transactions offer much higher bandwidth,being able to transfer multiple words in a single transaction,each word taking one cycle.Ignoring startup overheads,a25MHz TURBOchannel has a theoretical throughput of100megabytes/second.The TURBOchannel operates on physical addresses.This greatly simplifies the design of option modules,moving the burden of address translation to software.Under Unix,performing DMA to non-contiguous 4k pages of user memory would require an address translation every1024cycles or once every 40.These considerations lead naturally to questions of kernel performance.1.3Computer System Measurement with PAMsWe present a number of performance measures each obtained with a specific reprogramming of3mint and appropriate driving software.All these results were obtain on a DECstation 5000/200running Ultrix4.2with a3mint board in TURBOchannel option slot2.While the results are of interest in themselves we avoid detailed analysis,instead choosing to focus on the range of techniques which this approach gives us.This is the main contribution of the work.2Basic Measurement Devices2.1A Simple TimerFor ourfirst measurements we program the gate array on3mint to be a32bit counter based on[Vui91].It is clocked off the25MHz TURBOchannel clock(Seefigure1).This counter starts at zero when the design is loaded and free runs,wrapping around to zero periodically. This presents no problem provided we are only interested in measuring the duration of events that last less than171seconds.TbC1Figure1:A Simple TimerIn user mode we continually read this counter from a tight loop.Whenever successive reads differ by more than a threshold of a few tens of cycles the process supposes it has just resumed after being preempted and records the duration of the preemption in a histogram.We presenttwo such histograms infigure2.The solid line show the execution of the measurement process on an otherwise idle machine for a period of14hours during which12968494preemptions were observed.The dashed line shows execution with one competing compute bound process and is gathered over one hour—this data has been scaled up by14to be compatible with the idle machine data).We observe that the idle system records approximately260preemptions per second,that65%last between20and25and all but5%take less than100.Almost all of this activity can be explained by the3906system clock on the DECstation5000/200, generating256interrupts per second.Such base level system activity consumes about15%2.2User Virtual and Real TimersAdding an extra stoppable,loadable counter to our previous design and modifying the kernel to restore and save this second counter on context entry and exit provides a high resolution timer that runs in process virtual time(seefigure3).Instruction level profiling systems such as pixie developed by MIPS computer systems[Pix86]assume a perfect memory MIPS is a Trademark of MIPS computer systemsmodel.Discrepancies between estimated cycle counts and actual counts obtained from a high resolution timer running in process virtual time can point to memory bottlenecks caused by cache or TLB misses [GoHe90].1TbC10Figure 3:A Combined User Virtual and Real Timer3Active Measurement Devices3.1Raising InterruptsOnly so much of the system can be observed from user mode.To obtain more details on interrupt handling we must modify the kernel.3mint is a fully fledged TURBOchannel device,as such it can raise interrupts.We modify the simple timer to raise an interrupt each time bit 21of the counter goes high—roughly 5times a second.We add a device driver to field these interrupts.On entry it stores the current time from 3mint into memory it shares with user code and increments a shared count of interrupts taken.By comparing the stored time to the next smaller time at which a bit 21transition could have occurred user code can determine the time from hardware interrupt to handler and store this in a histogram.Provided this time is never greater than 200no ambiguity results.Likewise it can determine the time taken from handler back to user code by reference to the value recorded by the kernel and the current time value.Initial observations of time to handler histogram showed some curious artifacts when viewed at single cycle resolution.The histogram contained plateaus about 10cycles long preceded by peaks seven times higher (figure 4).Careful inspection of the inner loop of the user process that recorded interrupts revealed an inner loop of 8instructions,one which issued an I/O read to the 3mint.The plateau represents interrupts that are raised at some time during the 10cycles that the processor stalled during the I/O read.The peak represents interrupts that are issued during the other loop instructions for which the processor can immediately start handling the exception.We modify the interrupt raising design to schedule an interrupt on the transition of bit 21and issue it when it believes the processor is not currently or about to issue an I/O read.ItFigure4:Portion of Histogram of Unsynchronized Interrupt Raising Design makes this deduction by observation of its SELECT input and by knowledge of the length of the loop that user code executes.Since the design exists only for this particular measuring application,such extreme customization is perfectly acceptable.When the interrupt isfinally issued,the3mint stores the current time in a dedicated register for later interrogation by the interrupt handler.This design appears infigure5.bit 21IntrSelTbC1Figure5:Synchronized Interrupt Raising DesignSynchronized interrupts show much clearer peaks in the histogram of exception to interrupt handler.In one run,spanning5days and processing2483109interrupts from3mint,40% took exactly183cycles.Figures from this run appear infigure6.Thefigure exhibits marked peaks every15cycles starting at183.Curiously,while all other runs exhibit the same15cycle peaks,the detailed structure varies.In the example cited the main peak is at183cycles with a secondary peak at228cycles,exactly45cycles later.In other runs the secondary peak may be 30cycles later,or,occasionally,the main peak may appear at198or213cycles.We believe these are cache artifacts.To test this hypothesis we record the entry time of each of the half dozen routines in the path from the general exception trap entry point to the device specific interrupt handler.At the lowest levels this is very sensitive code.At the trap entry point itself all but two processor registers contain data from process running at the time of the exception;data which must be saved before they can be reused.In all the routines we must avoid adding extra overhead that will perturb our measurements.Recalling that scattered I/O writes are much cheaper than reads we add a smallfifo to our3mint interrupt raising design.On writes to a region of its address space the3mint now pushes the29low bits of the timer plus3bits from the target address into thisfifo.Writing to thisfifo costs just two cycles and requires only one free register—that in which the target address is loaded.When the interrupt handler is reached the fifo contents are read and stored in the shared buffer for later analysis by a user process.The new design appears infigure7.Figure8shows results obtained with this design.We place six probes in the path to the interrupt handler.Except The earliest point we could add instructions after the exception entry point.Vec1The start of main interrupt vector called after setup of kernel stacks and differentiation of interrupts from other exceptions.Vec2Before saving user state.Vec3After save of user state.k1First C routine called,beginning of discrimination of interrupt type.k2Next level of interrupt discrimination.pam Our3mint specific interrupt handler where we gather up the results of the probes.bit 21IntrSelTbC1Figure7:Synchronized Interrupt Raising Design with Event Fifo As we get further down the interrupt path the variations in cycle counts increases in keeping with the predictions of our cache miss hypothesis.The early stages of the path are taken by all exceptions and are therefore more likely to be in cache.We also place a probe in the user TLB miss handler.The MIPS architecture handles user TLBs in software and provides a special exception to ensure in rapid handling.In our analysis we check that the path recorded in thefifo is that we expect.It is possible for instance the kernel may service a higher priority interrupts and then continue on to service the3mint interrupt.We observe evidence of unusual sequences in02%of the cases recorded.3.3Summary of Interrupt MeasurementsIn60%of cases TURBOchannel interrupts start to get service from their handler with7. Within14,98%have service.This variation appears to be due to cache misses.Much longer delays sometimes occur and are probably due to interrupts being masked while other critical operations are performed.Delays of over1have been observed.4TURBOchannel DMA4.1DMA ThroughputTURBOchannel limits DMA to relatively short bursts,this simplifies the design of the memory system and help to ensure fair service even withfixed priority scheduling.TURBOchannel guarantees to support DMA transfers of at least64words.The implementation on the DECstation5000/200supports bursts of up to128words.There is afixed overhead in starting a DMA that is a amortized over the length of the transfer.With3mint reprogrammed to perform long sequences of DMA at a user specified blocklength to contiguous memory,we obtain the throughput results offigure9.Our design does not approach DMA as aggressivelyFigure8:Cycle Count Histograms of Steps in the Path to Interrupt Handleras it might,it waits7cycles between repeated requests,nevertheless for a blocklength of128 itRunning our DMA design continuously while using interactive applications,wefind that qualitatively the system remains quite useable even in the face of such heavy traffic.However, at these rates a4096byte DMA(the system page size)takes45;about six times the interrupt latency and about one third the time to take and return from an interrupt.It seems unlikely that system will remain useable if interrupts are used to feed DMA on a per page basis.We are currently investigating alternatives.4.2DMA LatencyIn the previous design3mint was DMAing counter values.To make the results of DMA more interesting we performed onefinal reprogramming3mint making it an interface to a Videk high resolution digital camera[Vide90](seefigure10).The Videk Camera outputs a1.5megabyte image at afixed rate of10megabytes/second. The data rate,while is well within the capabilities of TURBOchannel DMA,is difficult to handle by other means.Our3mint design buffers Camera output and DMAs it directly into system memory.In fact the Xilinx3000series programmable gate arrays are rather poor at implementing memory so we are only able tofit12words of buffering in the design.In order not to lose data we needed have a DMA request satisfied once every120cycles.The design detects these overruns and inserts black pixels for the lost data.Wefind in practice that the images are usually transfered very reliably,but occasionally we experience a lenghty burst of overruns.We discovered that long runs of uncached instruction reads without intervening writes would reliably cause the overruns.Occasionally the kernel would schedule such an instruction stream in the middle of an image transfer.DMA controlCamera DataReformat Pseudo ROM TbCCamera DataFigure 10:Videk Camera InterfaceReverting to our original DMA design we DMA the counter into system memory and record the time differences between the end and start of successive blocks in a histogram.We run the test both on an idle system and on a system with large disk copies in progress.The histogram appears in figure 11.With large copies active we observe much greater latency than on the idle system,however no DMA request is observed to be delayed for longer than 213cycles and there is a significant peak at exactly 195cycles.195is the DRAM refresh interval of the DECstation 5000/200.Apparently events associated with this activity cause the pending DMA to be granted.Unfortunately on 3mint we are just a little short of the neccessary storage to ensure lossless buffering.Nevertheless,3mint provides a workable interface to the videk camera—a quite unexpected application of the 3mint board.In figure 12we see the 3mint milliseconds before it DMAed the pixels in this image into the memory of a DECstation 5000/200.The 3mint is to the right,what lies between it and the camera is the board (on the left)which does level conversion.5ConclusionsPAM prove to be remarkably versatile at gathering low-level system measurements.These measurements would be very difficult to obtain by other means.They give unexpected insights into the foundation of a computer system and the lowest levels of the operating system.Im practical terms we have new data on interrupt handling under Unix and guidance in developing a peripheral that will tax the limits of the DECstation 5000/200’s memory system.And suprisingly we have a camera interface too.12510205010020050010002000500010000200005000010000020000050000010000002000000500000010000000250050100150200Figure 11:Histogram of DMA Latency under Different WorkloadsFigure 12:3mint Taking its own Photo6AcknowledgementsJean Vuillemin is the father of and force behind the PAM project.As part of our next generation PAM,Jean Vuillemin,Patrice Bertin and Didier Roncin designed and built the3mint board on which this work depends.7ReferencesBKW90A.Borg,R.E.Kessler,D.W.Wall,Generation and Analysis of Very Long Address Traces,Proc17th Annual Symposium on Computer Architecture,Seattle Washington 1990,pp.270-279.Wall91D.W.Wall,Systems for Late Code Modification,WRL Technical Note TN-19,Digital Western Research Laboratory,Palo Alto CA,1991.CBK88D.W.Clark,P.J.Bannon,and J.B.Keller,Measuring VAX8800Performance witha histogram Hardware Monitor Proc15th Annual Intl.Symposium on ComputerArchitecture,Honolulu,May1988,pp176-185.EC84J.S.Emer,and D.W.Clark,A Characterization of Processor Performance in the VAX-11/780Proc11th Annual Intl.Symposium on Computer Architecture,Ann Arbor MI,May1984,pp176-185.BRV89P.Bertin,D.Roncin,J.Vuillemin,Introduction to Programmable Active Memories, in Systolic Array Processors edited by J.McCanny,J.McWhirter and E.Swartzlander, Prentice Hall,pp.301-309,1989.Also available as PRL report3,Digital Equipment Corp.,Paris Research Laboratory,85,Av.Victor Hugo.92563Rueil-Malmaison Cedex, France.Xi89Xilinx,The Programmable Gate Array Data Book,Product Briefs,Xilinx,Inc.,1989.SBV89M.Shand,P.Bertin,J.Vuillemin,Hardware speedups in long integer multiplication, Proceedings2nd Annual ACM Symposium on Parallel Algorithms and Architectures.ACM Press,1990,pp138-145.TbC91Digital Equipment Corporation,TURBOchannel Hardware Specification,EK-369AA-OD-007,Digital Equipment Corporation,Maynard,Massachusetts,April1991.Vui91J.Vuillemin,Constant Time Arbitrary Length Synchronous Binary Counters,10th IEEE Symposium on Computer Arithmetic,Grenoble,France,June1991.Pix86MIPS Computer Systems,Language Programmer’s Guide,1986.GoHe90A.Goldberg,J.Hennessy,MTOOL:A Method For Detecting Memory Bottlenecks, WRL Technical Note TN-17,Digital Western Research Laboratory,Palo Alto CA,1991. Vide90Videk,VIDEK Megaplus Camera Operator Manual,Publication No.K00786, Videk,A Kodak Company,February1990.。