A Comprehensive Survey of Hybrid Design Verification Techniques

A Comprehensive Survey of Hybrid Design Verification TechniquesMackenzie Ryan ScottDept.of Electrical and Computer EngineeringUniversity of California,Davismrscott@June8,2005AbstractTraditional verification by simulation has proven to be less effective given more complex and modern designs.Many recently developed formal techniques,although powerful,suffer from an inability to scale with design size.Hybrid techniques attempt to combine formal verification with simulation to overcome these inherent difficulties.This survey seeks to provide an introductory overview of current hybrid techniques coupled with complex analysisto evaluate the feasibility and effectiveness of these techniques for future hardware design.First a background onsimulation and formal verification is given,and then a categorization of hybrid techniques is presented.Five differenthybrid approaches are put forth with analysis,andfinally,conclusions and future directions for hybrid techniques arediscussed.1IntroductionAs technology advances,designs are becoming more complex.Current technology enables millions of transistors tofit on a single chip,and designers are taking advantage of this in many innovating ways.However,for everynew design introduced,there are also new errors to discover.This has led to the development of the testing andverification researchfield.Academia and industry have exerted much effort into creating state of the art methods andtools to verify the functionality of new designs.Yet with all this effort,current designs are escalating in complexityfar faster than the tools used to verify them.It has been estimated that more than half of the design cycle is devotedto verification[15].All this has led to a verification dilemma of rising importance.1.1BackgroundIn past decades,semi-exhaustive simulation has been the main technique available to designers to best test their designs.Its ease of use and fast response have been invaluable for the act of”falsification,”orfinding as manydesign bugs as quickly as possible.However with increasing design size,designers have seen the effectiveness ofsimulation decrease rapidly.Nowadays,exhaustive simulation to prove the functionality of designs is more often thannot infeasible,allowing many costly errors to go undetected through the design process.This has caused the recentresurgence of interest in formal verification techniques,which were previously only practiced in academia.Formalverification is capable of a full100%design coverage,decidedly proving that a design will work or fail for everypossible input sequence.Nevertheless,many formal techniques suffer from a low automation level,a steep learningcurve,and the”state explosion problem,”where the state space grows exponentially as designs scale bigger,causingexponentially more memory and CPU time to be required for validation techniques[16].1.2Hybrid VerificationHybrid verification approaches attempt to combine traditional simulation with their more powerful formal coun-terparts.This is in the hopes that doing so will integrate the best characteristics of both categories,while alleviatingtheir weaknesses.This can allow greater and more efficient design coverage,leaving a more error free design.Hy-brid techniques are often termed”semi-formal”in the literature and often enable a tradeoff between accuracy ofverification,and acceleration of simulation[5].One of the big challenges to research in this area is the multitude of possibilities available to researchers.There are many formal verification methods as well as simulation tools and even more ways to combine them.But perhaps the greatest challenge lies in their ability to integrate these hybrid methods into the standard design process.Recently more commercial tools are becoming available which use hybrid methods.Promising examples include Magellan [13],ESP-CV[10],and Verifier[1].1.3Previous WorkA loose but generally accepted categorization of hybrid techniques has been proposed by Dill[5]which creates a spectrum of hybrid designs between simulation and formal verification.Starting closest to simulation,those cate-gories include Coverage Measures,Smart Simulation,Wide Simulation,and closest to formal verification,Prioritized Model Checking(figure1).Coverage measures are coupled with simulation to provide information on how thoroughly a design has been tested.Pseudo-random simulation can then be guided based on measures which include state and/or edge traversal, code coverage,signal toggle coverage,fault coverage,etc...More involved is smart simulation,which creates func-tional tests based on coverage metrics,and can be done either statically or dynamically as a design is being tested. Wide simulation chooses to symbolically represents a large set of states in a relatively few number of simulations. Traditional symbolic simulation is a prime example of this category[7].Lastly,prioritized model checking attempts to explore the state space in pieces,while using heuristics tofind design errors earlier.There also exist techniques which take ideas from multiple categories,and are very promising.Symbolic trajectory evaluation can be seen as an extension to model checking with roots embedded in a wide symbolic simulation.Its weakness has previously been only allowing verification offinite time intervals,however,current research has extended this method to more generalized,infinite time intervals[17].Figure1:Spectrum of verification techniques2Hybrid TechniquesWith a large set of possible techniques,this survey attempts to explore a representative set with methods from all of the above mentioned categories.The following arefive innovative approaches to the previously defined problems.2.1PartitioningAfirst fairly easy way to combine simulation with formal verification is to not combine them at all,but instead to judiciously use one or the other depending on the circumstances.This is the overall idea behind partitioning.Instead of using a single method to verify a complete design,it is possible to split a design into many smaller parts and verify those parts separately with any multitude of methods.This can include full formal verification or exhaustive simulation to completely verify the individual parts.This is in the hopes that verifying each part individually will correspond to a more error free design as a whole.The crux of this formulation is embedded in coverage measures for the different methods.The Integrated Design Validation(IDV)system[11]is a perfect specimen of partitioning.The main contributions of the system are a complexity analyzer and a partitioner.When given a design specification,the complexity analyzer works with the partitioner to split the design into smaller,more easily verifiable and less complex parts.Once this is done,it determines the”best”verification method for each piece and passes that piece to the appropriate verification module.For example it is known that symbolic trajectory evaluation works well when verifying memory dominated designs[11].In IDVs repertoire of methods are Symbolic Trajectory Evaluation(STE),Symbolic Model Checking (SMV),an Equivalence Checker(SMU-EQ),the SMU Functional Simulator,and thefive valued simulator,Speed5.Once the separate pieces are verified to the best abilities of each method with time and/or memory constraints,the results are passed to a coverage analyzer.This analyzer computes the state/transition coverage of each piece in the design,and if the coverage is not satisfactory,the unsatisfactory piece or the complete design is reanalyzed and repartitioned into something that might achieve better coverage results.This usually results in smaller partitions enabling more complete testing.The design is again verified and coverage is again computed.This repeats until a satisfactory coverage is attained for each individual design partition.At this point,the pieces arefinally put together as a whole design and run under a fast functional simulator.This is to make sure that the design works together as a whole.The system has a way to then compute the coverage of the whole design taking into account the separate coverage of the individual parts,as well as the vectors tested for the complete design.Figure2:Architecture of the Integrated Design Verification System(IDV) Although this technique seems a rather simple idea,it appears to be encouraging given the results of preliminary simulations that the research group has run.As the system has not been fully automated,it will remain to be seen whether this methodology will be given considerable thought.Without full automation,it is highly unlikely this technique will be adopted into a standard designflow.A design could be repartitioned several times,with each time requiring user input,which will dramatically slow down the design time,as well as maintain a high level of knowledge required for operation.This methodology does have the advantage of easy modification and scalability.If a new formal method or simulator were found to be more effective than a verifier originally in the system,only minor modification of the complexity analyzer,partitioner,and coverage analyzer would be required to either replace the old technique or incorporate the new one.Also,if it were not known what technique works well for a specific design,it would only take the generation of one set of inputfiles to this system to evaluate the performance of several different methods.2.2Guided SimulationAn assortment of papers analyzed for this survey[3,6,8,9,12]were determined to be some form of loosely guided simulation.Guided simulation is essentially a smart simulation technique which comes up with intelligent simulation vectors to run using various formal methods or coverage statistics.For example,Ganai et al.[6]used BDDs to generate vectors for nodes which are not excited by simulation,while Hazelhurst et al.[8]concentrated their efforts on using a new form of”hints”to guide state space search.The Ketchum tool[9]separates itself from these other methods by not only using a coverage statistic,but also integrating some unreachability analysis to focus test generation on reachable states.The Ketchum methodology was developed by a research group at Synopsys and is currently used inside their Magellan hybrid verification tool[13]. It uses a combination of random simulation,a SAT based bounded model checker,and symbolic simulation.The idea for combining these three methods is due to how they search the possible state space of a design.Randomsimulation tends to be able to search far and deep into the state space in a relatively short amount of time.This is in direct contrast to the SAT-based BMC which makes a very wide search of the state space close to the current initial state.This method also takes a very long time to execute.The symbolic simulator tends to be more robust than either method and so can search a wider state space than random simulation,and deeper than the SAT-based BMC,however it does have the constraint of using a large amount of memory and processing power to compute and store the BDDs for the symbolic formulas.Ketchum primarily uses the random simulator to explore the state space.This has the advantage offinding unexplored states rather quickly.Once the rate offinding unexplored states drops below a predefined threshold, Ketchum starts the SAT-based BMC.This methodically searches for a new state until either a memory or a time limit is reached.If this happens before a new state is found,then the symbolic simulator is started.Once a new state is found,then the generated vector tofind that state is re-simulated to bring the design to that state.At this point,the random simulation is started again with the impression that this is a new unexplored area of the state space so it will be easy for the random simulator tofind new states.A visualization of this method is shown infigure3.Figure3:Visualization of how Ketchum explores the state space The main strength of Ketchum is that it has been shown to successfully verify extremely large designs relative to other verification methods(4500latches,170K gates).As well as large designs,it also is much more effective than random simulation,reaching up to6x more coverage states in experimental results.Since it is basically a simulator, it should not have as many problems scaling with design size when dealing with total execution time.However,the effectiveness will decrease because the formal methods are still essentially unmodified.If the random simulation takes the design to a state which is not anywhere close to important unreached states,even the symbolic simulator won’t be robust enough tofind a new state in a short amount of time or with a small amount of memory.2.3Saturated SimulationThe notion of Saturated simulation is put forth by Yuan et al.[18]and refined by Aziz,Kukula,and Shiple[2], specifically for the problem of interacting controllers.The idea behind saturated simulation is to simulate as many reachable control states in every cycle as is computationally feasible,thus creating a wide simulation.Saturated simulation was mainly developed to tackle the problem of invariant checking(making sure all reachable states are valid states),also commonly known as invariant verification or assertion checking.By simulating many possible control states at once,it is possible to check many more states for whether they lie in the invariant or not,compared to random simulation.Image and pre-image operations can be computed directly using BDDs,and are the basis of saturated simulation. Given a set of initial states,an image is computed which shows the set of states which can be entered from the set of initial states in a single cycle.The initial states are then OR-ed with the image,and projected to a new set of states. The projection operation basically attempts to trim off as much data as possible,while still leaving a representative set of all the possible control states.The control states are”saturated”in this sense.After projection,this set is checked against for intersection with the complement of the invariant(illegal states).With no intersection present,the current set of states becomes the new initial set,and the algorithm repeats.By saturating only the control bits,they hope to avoid or mitigate the state explosion problem.Experimental results show that this method outperforms traditional cycle simulation without control saturation in terms of the number of states,edges,and depth of the design reached.This suggests that this is indeed an improvement on the existing standard method.Although specifically developed for invariant checking,it seems as though it should be applicable to other veri-fication problems as well.With BDDs already generated,they could be checked for correctness of operation beforeFigure4:Simple example of saturated simulation in actionthey are projected and cut.This would be important since most design errors are found in the control portion of a design.While saturated simulation attempts to address the state explosion problem by only using a representative set of control states,it does not guarantee there will not be a problem.Although most designs have several times more storage elements dedicated to data instead of controlflow,large designs such as a microprocessor or anything with a complexflow of operations resulting in many different control states could potentially be a problem.If verifying a whole system with many interacting controllers,there might be several times the number of bits dedicated to control, coupled with the fact that extra bits incur an exponential increase in the number of states,leads to the same state explosion problem which the creators attempted to avoid.2.4Retrograde AnalysisRetrograde analysis is yet another technique specifically addressing invariant checking.When using simulation for invariant checking,it can be hard to generate an input/sequence which will get the design into an illegal state since there can be many conditions which need to be satisfied.The ultimate goal is to provide simulation with a larger and easier”target”for test vectors to hit[18].With this method,formal methods are used to allow more efficient simulation.The main idea in retrograde analysis is to work backwards from the endpoint,or in this case,an illegal set of states.Pre-images,which are similar to the images discussed above in saturated simulation,are computed for a user specified number of cycles,or until a memory or time constraint is met.Pre-image i generates the set of states which can reach thefinal set of states in i cycles,in the form of symbolic BDDs.At the end,a large set of states is generated which can theoretically reach the set of illegal states.At this point random simulation is used in a normal fashion starting from a good set of initial states.However,when a random simulation takes the design into a state which is part of the previously computed pre-imaged state set,an error in the design is found.This is true because it is now possible for the design to get to the set of illegal states given the proper set of test vectors.Besides computing pre-images,it is also possible to compute images of the initial states to help close the gap in the simulation space between the set of good initial states and the set of illegal states.This requires picking an initial state in the set which is”close”to the illegal target states.Yaun et al.[18].propose using the Hamming distance of the state vectors as a measure of closeness.This can also be recursively applied using forward reachability with images, randomly simulating some vectors,and then more images until the target states are reached.These three versions of retrograde analysis techniques are illustrated infigure5.Experimental results show an almost exponentially decreasing number of random test vectors are needed with each successive pre-image.However,the BDD size required and number of target states increase exponentially as well,possibly countering the potency of this method for large designs.Hamming distance appears to be a good closeness metric in most cases,yet care should be taken to analyze why this is so.Cases could be possible when the data elements do not matter and only the control state elements should be considered.Because the data elements usually outnumber the control ones,the hamming distance metric could oppose what would actually be a”close”state.In this case,abstracting out the data bits would be beneficial.Although not mentioned in the papers surveyed,retrograde analysis has the advantage that besides helping to verify invariant checking,it can also verify the total design at the same time with coverage statistics stemming from the simulation vectors.It is also veryflexible.This approach works with normal simulation,but it might also beFigure5:Visualization of how retrograde analysis can enable a larger state space for the target of simulation amenable for extension to a form of symbolic simulation as well.Then not only would there be a larger target for simulation,but more vectors would be traced at once providing a higher chance offinding a path to an illegal state. 2.5Semi-Formal Bounded Model CheckingSemi-formal bounded model checking is an approach to verification that is almost identical to regular bounded model checking.In a model checking approach,a specification is written as several temporal logic formulas,and the design being verified is checked for whether it satisfies those formulas[15].Bounded model checking(BMC) reduces the problem of temporal logic model checking into a problem offinding satisfying inputs for a combinational circuit.The combinational circuit is created by unrolling a sequential circuit as shown infigure6.This can be done because with BMC,the logic equations are bounded by a time limit.Normally,the combinational circuit is converted into conjunctive normal form(CNF)(figure7)and handed to a SAT solver tofind the satisfying input.However in the semi-formal approach,instead of a SAT solver,the circuit is compiled for simulation and vectors are explicitly tested for satisfiability[4].Ultimately this method is an example of using simulation to enhance an existing formal method,rather than previous approaches in this survey which mainly try to use formal methods to generate a smarter simulator.Figure6:Unrolling a sequential circuit into a combinational one(b)CNF Equation(a)ExampleCircuitFigure7:Transforming a circuit into CNFThe main algorithm employed is a brute force tester shown infigure8which simply tries all the possible values and returns either SATISFIABLE or UNSATISFIABLE if itfinds a vector or not.while(vector=choose_an_untried_vector()){set_inputs(vector);//Assign vector to circuit inputssimulate_circuit(k);//Simulate time0through kif(circuit.f.value==TRUE)return SATISFIABLE;record_unsuccessful_trial(vector,...);}return UNSATISFIABLE;Figure8:Brute force simulation algorithmThe main advantage of this approach is that a circuit compiled into machine code enables very fast simulation, which can enable this approach to be much faster than a SAT solver based method depending on the circuit being simulated.It also is able to simulate circuits containing non-boolean elements(ex:tristate buffers)without the overhead of encoding more equations[4].However a SAT solver does have some advantages over random simulation. First is that a simulator has an exponentially increasing number of simulation vectors which it might have to run to come up with a result.Also,a SAT solver can be a lot smarter than a simulator,choosing to avoid assigning values to variables which will always make the resulting satisfiability equation false,as well as choosing smart variables on which to perform case-splitting.Semi-formal BMC attempts to alleviate these problems with what is called”skipping cubes.”Skipping cubes ends up generating the ability to backtrack and introduce a learning ability to the simulator.With some fancy mathematical rigor,Bingham and Hu[4]develop skipping cubes so that the simulator knows what input variables effect the value of f k,and can henceforth backtrack to the relevant point in simulation instead of just the previous decision.This effectively prunes the search space into a more manageable size.When comparing this semi-formal method to a leading free SAT solver for BMC,Z-Chaff,experimental results indicate that this method is highly competitive when looking at the full verification times without including the compilation time.With this time included,it is only competitive for large circuits,which is why this method is designed,so small circuit results don’t matter much in a comparison anyways.A huge advantage of this method is that because it simulates vectors,it can still give a coverage result when the method times out on extremely large circuits.A normal SAT solver will not give any valuable information except that it couldn’tfind values in the given time.Although this is a good combination of simulation and model checking,it still suffers from the classical drawbacks of model checking since it still requires very knowledgeable designers to write the original temporal logic formulas describing a system.The simulation part will scale with designs as far as speed is concerned.However given a large design,it is not clear that even the backtracking and learning ability created with skipping cubes will be able to narrow down the search space to little less than a semi-exhaustive search.Creating the hope for a chance of”getting lucky”with a randomly tested vector.2.6Discussion and ComparisonBecause there are no solid standards,and even a lack of real test circuits for verification,a fair comparison is difficult to prepare.So instead,this survey will evaluate the relative strengths and weaknesses of the specified methods to each other in terms of scalability,effectiveness,and ease of incorporation into existing designflows.Coverage measures will continue to be used for as long as verification exists,so it is only natural that they be used in more a intelligent manner than simply evaluating an end result.To this end,the partitioning method and general guided simulation are good hybrid approaches which can be used in conjunction with other methods.These two also stand a good chance of being assimilated into an establishedflow due to their similarity with existing industrial verification methods,and the ease of implementation.These approaches,along with retrograde analysis,provide the most scalability of the surveyed techniques since they currently do not rely on formal methods for the verification of the whole system,but rather a tiny portion.Retrograde analysis is also an exciting prospect in its own right,as it tends to be orthogonal to most other ap-proaches presented in this survey.Some more work might be done to apply a smarter guided approach to the simula-tion,and incorporate the ideas into other methods.Saturated simulation and other wide simulation techniques which include the use of BDDs should not be ruled out as possibilities for verification.Sudden advances in memory capacity,processing power,and novel data structures could suddenly make this an attractive option once again because of the available efficiency.However,for now,it does not seem like these methods will scale unless significant work is done to stem the state explosion problem.Model checking methods,suffer from the weakness of the user having to originally write the temporal formulas used.These can be misinterpreted,or some properties might never be written and instead taken for granted[16].The semi-formal model checking method discussed earlier,does not address any of these issues,but instead concentrates on how to speed up the execution.Until these fundamental issues are addressed,or the process of creating formulas can be automated from an HDL design,model checking approaches will prove to be sub-par compared with other methods.3ConclusionsExisting formal methods haveflaws and limitations which keep them from being integrated into the product designflow,despite their proven effectiveness with small designs.All the while simulation is steadily decreasing as an effective tool to verify complex designs.Several hybrid methods have been shown to be a valid alternative solution to the existing verification methodologies.They can overcome the inherent limitations by subtly trading off full coverage for ease and quickness of design validation.3.1Future Directions in Hybrid VerificationAlthough no current formal verification technique exists today which can be used to solve all verification prob-lems,hybrid verification techniques can be seen as a stepping stone on the way to a mainstream formal verification methodology.The current goal of semi-formal methods should be to complement existing designflow tools and increase the acceptance of formal methods in industry.Often times,many semi-formal methods give no reason as to why they can’t be used concurrently.Future research might not only want to focus on how to integrate formal methods with simulation,but also how best to integrate previously published hybrid methods with each other to pro-duce even more effective solutions.Future research might also not be as involved with verifying designs,but more with using verification to tolerate failures.A novel approach developed by Mneimneh et al.[14],couples a”checker”processor with a complex microprocessor in an at speed emulation methodology,noting that much of the design(eg: branch prediction)is fault tolerant.The checker processor is formally verified,and checks the core processor for faulty execution,stalling and re-executing the core processor as needed.As seen in this survey,many obvious approaches have not yet been tried.And given the relatively recent prob-lems and sparked interest in verification,there is much room available for the research and development of hybrid verification techniques.References[1]Verifier by a HDL,/product verifier.html,2005.[2] A.Aziz,J.Kukula,T.Shiple,Hybrid Verification Using Saturated Simulation,DAC,p.615-618,1998.[3]J.P.Bergmann,M.A.Horowitz,Improving Coverage Analysis and Test Generation for Large Designs,CAD,p.580-583,1999.[4]JD.Bingham,AJ.Hu,Semi-formal Bounded Model Checking,CA V,p.280-294,2002.[5] D.L.Dill,What’s Between Simulation and Formal Verification?,DAC,p.328-329,1998.[6]M.K.Ganai,A.Aniz,A.Kuehlmann,Enhancing Simulation with BDDs and ATPG,DAC,p.385-390,1999.[7]M.Gordon,/users/mjcg/Research00/,2000.[8]S.Hazelhurst,G.Kamhi,O.Weissberg,L.Fix,A Hybrid Verification Approach:Getting Deep into the De-sign,DAC,p.111-116,2002.[9]PH.Ho,T.Shiple,K.Harer,J.Kukula,R.Damiano,V.Bertacco,J.Taylor,J.Long,Smart Simulation UsingCollaborative Formal and Simulation Engines,ICCAD,p.120-126,2000.[10]ESP-CV formerly by Innologic,/products/acmgr/innlo/innloacq.html,2004.。