Speeding-up Fault Injection Campaigns in VHDL models

Speeding-up Fault Injection Campaigns in VHDL modelsB. Parrotta, M. Rebaudengo, M. Sonza Reorda, M. ViolantePolitecnico di Torino, Dip. di Automatica e Informatica, C.so Duca degli Abruzzi 24,I-10129 Torino, Italy{parrotta, reba, sonza, violante}@polito.itAbstract. Simulation-based Fault Injection in VHDL descriptions is increas-ingly common due to the popularity of top-down design flows exploiting thislanguage. This paper presents some techniques for reducing the time to per-form the required simulation experiments. Static and dynamic methods areproposed to analyze the list of faults to be injected, removing faults as soonas their behavior is known. Common features available in most VHDL simu-lation environments are also exploited. Experimental results show that theproposed techniques are able to reduce the time required by a typical FaultInjection campaign by a factor ranging from 43.9% to 96.6%.1 IntroductionIn recent years, there has been a rapid increase in the use of computer-based sys-tems in areas such as railway traffic control, aircraft flight, telecommunications, and others, where failures can cost lives and/or money. This trend has led to concerns regarding the validation of the fault tolerance properties of these systems and the evaluation of their reliability.On the other side, the continuous increase in the integration level of electronic systems is making more difficult than ever to guarantee an acceptable degree of reliability, due to the occurrence of un-modeled faults and soft errors that can dra-matically affect the behavior of the system. As an example, the decrease in the magnitude of the electric charges used to carry and store information is seriously raising the probability that alpha particles and neutrons hitting the circuit could introduce errors in its behavior (often modeled as Single Upset Errors) [1].To face the above issues, mechanisms are required to increase the robustness of electronic devices and systems with respect to possible errors occurring during their normal function, and for these reasons on-line testing is now becoming a major area of research. No matter the level these mechanisms work at (hardware, system soft-ware, or application software), there is a need for techniques and methods to debug and verify their correct design and implementation.Fault Injection [2] imposed itself as a viable solution to the above problems. Sev-eral Fault Injection techniques have been proposed and practically experimented; they can basically be grouped into simulation-based techniques (e.g., [3][4]), soft-ware-implemented techniques (e.g., [5][6][7][8]), and hardware-based techniques (e.g., [9][10]).As pointed out in [2] physical fault injection (hardware- and software-implemented fault injection approaches) is more suited when a prototype of the system is already available, or when the system itself is too large to be modeled and simulated at an acceptable cost. Conversely, simulation-based fault injection is very effective in allowing early analysis of designed systems, since it can be exploited when a prototype is not yet available. The main disadvantage of simulation-based fault injection technique versus hardware- and software- ones is the high CPU time required to simulate the model of the system.In the last years, VHDL has become a de-facto standard for describing systems in particular when automatic synthesis tools are exploited; therefore, several research-ers have addressed the problem of how to perform fault injection campaigns on VHDL descriptions [3] [4]. These approaches are simulator-independent, i.e., injec-tion is performed through the adoption of special data types or fragments of VHDL code that are dedicated to modify the system behavior. They do not rely on simula-tor features, nor require the modification of simulation tools, therefore can easily fit on any design flow.The approaches proposed so far neglect the problem of reducing the time spent running fault injection campaigns. To deal with complex system new methods should be devised to minimize the time spent during simulations.In this paper we address the problem of improving the efficiency of simulation-based fault injection and we show how simulation time can be dramatically short-ened. The technique we developed analyzes the faults that will be injected in order to identify fault effects as early as possible and exploits the features provided by modern commercial VHDL simulators to speed-up injection operations. Our ap-proach comprises three steps:1.golden run executions: the system is simulated without injecting any faults anda trace file is produced, recording information on the system behavior. Moreo-ver, the state of the simulator is sampled and stored allowing to resume the simulation at a given time instant;2.static fault analysis: given an initial fault list, by exploiting the informationgathered during golden run computation, we identify those faults which effects on the system are known a-priori, and remove them from the fault list. This step exploits a sub-set of already proposed fault collapsing rules [11];3.dynamic fault analysis: during fault injection, the behavior of the system is pe-riodically compared with the golden run. The simulation is stopped as early as the effect of the fault on the system becomes known, i.e., either the fault alters the system behavior or the fault does not produce any effect.We exploit standard commands offered by commercial VHDL simulators for in-jecting faults and tracing system behavior. This technique minimizes the intrusive-ness of our approach and, by exploiting standard commands that are usually pro-vided by modern VHDL simulators, it is of general use.To assess the effectiveness of our technique, we performed fault injection ex-periments on a benchmark target system composed by a single microcontroller. Weanalyzed the behavior of several programs, and we recorded a speed-up ranging from 43.9% up to 96.6%.The remainder of this paper is organized as follows. Section 2 describes the sys-tem architecture where faults are injected. Section 3 describes the techniques we developed, while Section 4 reports some preliminary results assessing their effec-tiveness. Finally, Section 5 draws some conclusions.2 The target architectureThe system we considered is composed of an Intel i8051 microcontroller equipped with an external RAM memory storing the code the i8051 executes. All the programs we have considered have a data segment that fits in the i8051 internal memory composed of 128 bytes of static RAM; therefore our target architecture does not exploit an external RAM for storing data. The whole system is described in VHDL at the Register Transfer (RT) level.The fault model we are considering is the transient single bit flip. We injected faults in the following system components:•i8051 control registers and internal SRAM•external RAM storing the code.This assumption allows us to evaluate the effects of single transient faults both in the data or in the code.3 Proposed approachThe purpose of our approach is to minimize the time required for performing fault injection campaigns. We addressed this problem by performing fault analysis (before starting and during the fault injection campaign) and resorting to simulator commands that can be used to quickly reach injection time. The proposed approach is based on the assumption that the simulation runs are deterministic, i.e. for a given set of inputs there is only one trace of the system golden execution.To implement this approach the fault injection tool cooperates with a standard VHDL simulator, as Figure 1 shows. The fault injection manager issues commands (such as run commands or query commands) to the simulation back-end, which in turn sends responses to the Fault Injection manager.In the following subsections we will detail how our technique can be imple-mented; in particular, subsection 3.1 describes the information we gather on the behavior of the system during golden run execution, then subsection 3.2 shortly recalls the fault-collapsing rules we exploited and subsection 3.3 details how dy-namic fault analysis can be implemented. Finally, subsection 3.4 describes a tech-nique which exploits simulator facilities and that allows to save the most of the time required to reach injection time.Fig. 1. Fault injection tool overview3.1 Golden run executionThe purpose of this step is to gather information on the behavior of the system: given a set of input stimuli that will remain constant in the following fault injection campaigns, two sets of information are gathered, one for performing static fault analysis and one for performing dynamic fault analysis.Static fault analysis requires:•data accesses: whenever a data is accessed, the time, the type of access (read or write) and the address are stored;•register accesses: whenever a register is accessed, the time, the register name and the type of access are stored;•code accesses: at each instruction fetch, the address of the fetched instruction is stored in a trace file;We collect the needed information resorting to ad-hoc modules written in VHDL, called code/data watchers, that we insert in the system model (Figure 2). This ap-proach is not intrusive, since code/data watchers work in parallel with the system and do not affect its behavior.Conversely, for performing dynamic fault analysis we sample the state of the system with a given sampling frequency. The state of the system reports the content of the processor registers and the content of the data memory. We gather this infor-mation by periodically stopping the simulation and observing the system state re-sorting to simulator commands. The values gathered during sampling are stored in main memory, where we record the time at which the sample is taken, the contents of processor control registers and the content of the processor memory.This approach is very simple and effective because allows to capture snapshots of the system with zero intrusiveness. On the other hand, when addressing very large systems, it could require the availability of large amounts of both memory and disk space. As a consequence, the sampling frequency should be carefully selected.3.2 Static fault analysisFaults are removed from an initial fault list according to two sets of rules; one for faults intended to be injected in the data and one for faults in the code. The rules areapplied by analyzing the information gathered during golden run execution.We remove from the fault list a fault affecting data if it verifies at least one of the following conditions:1.given a fault f to be injected at time T at address A, remove f from the fault list ifA is no longer read after time T: this rule allows removing the faults that:−do not affect the system behavior, or−remain latent at the end of the simulation;2.given a fault f to be injected at time T at address A, remove f from the fault list ifthe very first operation that involves A after time T is a write operation.Conversely, we remove a fault in the code if it verifies the following condition: given a fault f to be injected at time T at address A, remove f from the fault list if the address A corresponds to an instruction that is no longer fetched after time T. This rule identifies faults that do not produce any effect and whose injection is therefore useless.Fig. 2. Modified system architecture3.3 Dynamic fault analysisDynamic fault analysis is based on the idea of identifying as early as possible the effect of the injected fault.As soon as the effect of a fault become evident, we stop the simulation, poten-tially saving a significant amount of simulation time.The fault injection routine we exploit is described in Figure 3.Fig. 3. Fault injection routineThe fault injection routine starts by settings a set of breakpoints in the VHDL code of the system. Breakpoints are a feature of modern VHDL simulators that of-fers a simple mechanism to capture the following situations:1.program completion: a breakpoint is set so that simulation is stopped after theexecution of the last instruction of the program running on the system;2.interrupt: in order to detect asynchronous events, a breakpoint is set to the VHDLstatements implementing the interrupt mechanism activation;3.time out: the simulation is started with a simulation time higher than the timerequired for program completion. A time out condition is detected if simulation ends and no breakpoint has been reached.After all the required breakpoints have been properly set, we simulate the system up to the injection time, then we inject the fault. Injection is done by exploiting the VHDL simulator commands to modify signals/variables in the VHDL source. After injection we start a loop where the system is simulated up to a sample point and the state of the system is compared with the information gathered during golden run execution.At each sample point, we check for the following situations:1.no failure: the state of the system is equal to the state of the golden run at thesame time, this implies that the fault effects disappeared from the system and that the fault has no effect on the system behavior; as a consequence, the simulation can be stopped;2.the state of the system does not match the golden run; in this case two alterna-tives are available:−failure: the fault has affected system outputs, therefore its effect on the system is known, and simulation can be stopped;−latent fault: the fault is still present in the system but it has not affected system outputs, and therefore further simulation is still required.3.4 Checkpoint-based optimizationsModern VHDL simulators offer several features that can be exploited to effec-tively reduce simulation time of fault injection campaigns. In particular, we ex-ploited the possibility of storing simulator state and then restoring it during simula-tions. The rationale behind this approach is shown in Figure 4.Fig. 4. Simulator-dependent optimizationGiven a fault f to be injected at time T , a not-optimized fault injection tool will spend a time equal to T setup to reach injection time. Conversely, during golden run execution, we sample the status of the simulator (and thus the status of the system under analysis) with a given frequency, and we store the information we obtain in a collection of system status files . Then, when fault f has to be injected, we resume the simulator from the first checkpoint before T (checkpoint 2 in the example of Figure 4 therefore, the time spent to reach injection time becomes T’’setup .Two conditions must be satisfied for this approach to be effective:1. let T R be the time for restoring the status of the system at a given checkpoint, the following inequality must hold:setup R setup T T T <+"(1)2. the size of the checkpoints collection should be kept as small as possible.As a consequence, the number of checkpoints should be carefully selected on the basis of both the time required for simulating the system and the size of the system.4 Experimental resultsTo evaluate the effectiveness of the proposed approach we wrote a prototypical implementation of the Fault Injection manager shown in Figure 1, which amounts to 790 lines of C code. During our experiments, we exploited Modeltech VSIM ver 5.1to run VHDL simulations.We wrote some simple benchmark programs and run them on the system archi-tecture described in Figure 2 which amounts to 6,700 lines of VHDL code (380 linesare devoted to implement CODE and DATA watchers). The benchmarks charac-teristics are summarized in Table 1.Table 1. Benchmarks descriptionFor each benchmark, we randomly selected 2,000 faults (1,000 in the data and 1,000 in the code) and we serially injected them in the system without resorting to any optimization. We then turned on dynamic fault analysis and checkpoint-based optimization and re-injected the selected faults. Finally, we collapsed the initial fault list resorting to static fault analysis and then injected the remaining faults. In the latter experiments dynamic fault analysis and checkpoint-based optimization are turned on. Results, gathered on a Sun SparcStation/5 running at 170 MHz and equipped with 128 Mbytes of RAM are reported in Table 2 and 3. Table 2 reports results gathered when injecting in the data, while Table 3 reports data gathered when injecting in the code. The columns have the following meaning:1.benchmark reports the name of the benchmark;2.T0 is the time required for injecting 1,000 faults (either in the data or in the code)when no optimizations are exploited;3.T1 is the time required by fault injection when dynamic fault analysis and simu-lator dependent optimization are exploited;4.T2 is the time required by fault injection when all the optimization previouslydescribed are exploited;5.1-T1/Tand 1-T2/Tare the speed-up we attain when the correspondent optimiza-tions are turned on.From the reported tables, we can observe that the complexity introduced in the Fault Injection manager to perform dynamic and static fault analysis and to exploit simulator facilities is worth paying, since it allows to speed-up simulated fault in-jection up to 96.6%. Moreover, static fault analysis plays a key role, since its adop-tions always double the attained speed-up.Our approach attains the best results when injecting faults in the data. This can be explained by considering that several faults exist that disappear from the system very soon after their injection (due to error detection and correction mechanisms or due to the system behavior) without affecting the system. Even if this faults cannot be eliminated by static analysis, the will be catch by dynamic analysis, thus saving a significant amount of CPU time.Conversely, when faults in the code are concerned, if a fault is injected because it fails static analysis, it remains latent during all the program execution. Therefore,simulation can be stopped only when fault effects reach system outputs or when the program ends.Table 2. Results on dataTable 3.Results on codeWe evaluated the performance of our approach when varying the number of checkpoints. We considered the BUBBLE benchmark and we run four fault injection campaigns where 100 faults are injected and respectively 5, 25, 50 and 100 check-points are used. We recorded the time and the disk occupation each campaign re-quired, and plotted them in Figure 5.Fig. 5. Space occupation versus timeAs this figure suggests, a trade-off exists between space required for storing checkpoints and the time for performing a fault injection campaign. For the bench-mark we considered, 25 checkpoints offer the best trade-off since the experiment requires approximately 7 Mbytes of disk space and injects 100 faults in 219 seconds.5 ConclusionsThis paper presented some techniques for reducing the time required for per-forming simulation-based fault injection campaigns. Static and dynamic methods are proposed to analyze the list of faults to be injected, removing faults as soon as their behavior is known. Common features available in most VHDL simulation environments are also exploited. Experimental results show that the proposed tech-niques are able to reduce the time required by a typical fault injection campaign by a factor ranging from 43.9% to 96.6%.References1. M. Nikoladis, Time Redundancy Based Soft-Error Tolerance to Rescue Nanometer Tech-nologies, IEEE 17th VLSI Test Symposium, April 1999, pp. 86-942.R. K. Iyer and D. Tang, Experimental Analysis of Computer System Dependability, Chap-ter 5 of Fault-Tolerant Computer System Design, D. K. Pradhan (ed.), Prentice Hall, 1996 3.E. Jenn, J. Arlat, M. Rimen, J. Ohlsson, J. Karlsson, Fault injection into VHDL Models:the MEFISTO Tool, Proc. FTCS-24, Austin (USA), 1994, pp. 66-754.T.A. Delong, B.W. Johnson, J.A. Profeta III, A Fault Injection Technique for VHDLBehavioral-Level Models, IEEE Design & Test of Computers, Winter 1996, pp. 24-33 5. G.A. Kanawati, N.A. Kanawati, J.A. Abraham, FERRARI: A Flexible Software-BasedFault and Error Injection System, IEEE Trans. on Computers, Vol 44, N. 2, February 1995, pp. 248-2606.J. Carreira, H. Madeira, J. Silva, Xception: Software Fault Injection and Monitoring inProcessor Functional Units, DCCA-5, Conference on Dependable Computing for Critical Applications, Urbana-Champaign, USA, September 1995, pp. 135-1497.S. Han, K.G. Shin, H.A. Rosenberg, Doctor: An Integrated Software Fault-InjectionEnvironment for Distributed Real-Time Systems, Proc. IEEE Int. Computer Performance and Dependability Symposium, 1995, pp. 204-2138.A. Benso, P. Prinetto, M. Rebaudengo, M. Sonza Reorda, EXFI: a low cost Fault Injec-tion System for embedded Microprocessor-based Boards, ACM Transactions on Design Automation of Electronic Systems, Vol. 3, Number 4, October 1998, pp. 626-6349. J. Arlat, M. Aguera, L. Amat, Y. Crouzet, J.C. Fabre, J.-C. Laprie, E. Martins, D. Powell,Fault Injection for Dependability Validation: A Methodology and some Applications, IEEE Transactions on Software Engineering, Vol. 16, No. 2, February 199010. J. Karlsson, P. Liden, P. Dahlgren, R. Johansson, U. Gunneflo, Using Heavy-Ion Radia-tion to Validate Fault-Handling Mechanisms, IEEE Micro, Vol. 14, No. 1, pp. 8-32, 1994 11. A. Benso, M. Rebaudengo, L. Impagliazzo, P. Marmo, Fault-List Collapsing for FaultInjection Experiments, RAMS'98: Annual Reliability and Maintainability Symposium, January 1998, pp. 383-388。