Automatic test case generation for critical embedded systems

Automatic test case generationfor critical embedded systemsWiels2Seguin2 Virginie Guy Durrieu2Odile Laurent1 Christel1Airbus 2ONERA / DTIM316 route de Bayonne 2 avenue E. Belin, BP 402531000 Toulouse cedex 03 France 31055 Toulouse cedex Franceurent@{durrieu,seguin,wiels}@cert.frAbstractThis paper presents a research project on the feasability of automatic test generation from formal specification in an industrial context. Airbus has used SCADE for several years to specify critical avionics systems. We have experimented automated test generation tools from SCADE specifications and defined a coverage criterion adapted to SCADE specifications.1. IntroductionAirbus has used formal methods for several years to specify avionics systems. Thanks to these techniques, development cycles have been shortened significantly, automatic code generation from formal specification played an essential role in this improvement. A first research project conducted by Airbus and ONERA showed that the use of formal techniques for the specification of avionics systems allowed the use of formal proof techniques for the validation of these systems [LMW01]. Now SCADE Prover is a commercial tool to verify properties of a formal SCADE specification. Static validation techniques can thus be used in an operational way and this constitutes a first breakthrough with respect to the classical verification and validation process which was exclusively based on dynamic techniques such as simulation and test. It is not however realistic to think that this kind of techniques will be able to replace all tests for critical avionics systems. This industrial fact leads us to look for an assistance in the generation of test cases using automatic test generation from formal specification.This paper presents the work done in a project (financed by DPAC, French Program Direction for Civil Aviation) where the objective was to study the feasibility of automatic test generation based on formal techniques by applying them to operational industrial systems. Section 1 describes the current verification and validation means. Section 2 presents our objectives with respect to these practices and explains the approach. Section 2 also presents the structural coverage criterion used in our approach. Section 3 presents the two tools we used and synthesizes the experimentations done with these tools. Conclusion and future work end the paper.2. Current verification and validation meansCurrently, the main verification and validation activities done at Airbus are the following:-System level simulation: the considered system is validated in a simulated environment. The designer validates the software specification on real time computers providing a panel of commands representing possible pilot actions.-Aircraft level simulation: several systems are validated in a simulated environment.-System benches: first tests with real equipments, on a single system.-Multi-systems benches: at this stage, real equipments exist for the different systems, the goal is to be sure they are correct with respect to their specification.-General bench: last tests before the tests on the real aircraft, they are done on real equipments and with a good representation of the aircraft environment.In this project, we were particularly interested by verification and validation activities done at the detailed specification level (formal SCADE specifications). For this kind of specifications, validation activities are possible before system level simulation: tests using SCADE simulator and proof of properties using SCADE Prover.All the above verification and validation activities necessitate the definition of pertinent test vectors (for simulation activities) and the definition of properties (for proof activities). But the definition of test cases is not easy because they have to ensure that all the functions of the system are tested and that dangerous configurations are not forgotten.Civil airworthiness authorities require, for systems of level A, B or C, functional verification and validation activities to ensure correct operation in normal and abnormal conditions, criteria for the termination of these functional V&V activities are also to be defined. To respect these requirements, Airbus defines tests by identifying -Equivalence classes: partitioning of inputs such as a test of a given class is functionally equivalent to any other test of the same class,-Singular points,-Structural coverage criteria, based on the structure of the specifications, chosen as termination criteria.Currently, functional test cases are described manually in test plans. For each functional test case, the following pieces of information are provided: -Textual description of the test objective,-Identification of minimal test cases (that must imperatively be executed),-For each test objective, the set of chosen scenario and expected results. Each scenario is described as a sequence of operations together with the associated means.3. SCADEThe SCADE environment, developed by Esterel Technologies, was defined to assist the development of critical embedded systems. This environment is composed of three tools:• a graphical editor,• a simulator,• a code generator that automatically translates graphical specifications into C code.The SCADE language is a graphical synchronous data flow specification language. In this language, time is divided into discrete instants defined by a global clock. At instant t the synchronous program reads inputs from external environment and computes outputs. The synchrony hypothesis states that the computation of the outputs is made at the same instant t . It means that for periodic systems like control/command systems, all outputs are computed at each cycle.A system is described by several interconnected nodes. A node computes values for its outputs from values of its inputs. A node is defined by a set of equations and assertions. An equation VAR=EXPR specifies that the variable VAR is always equal to EXPR . An assertion assert BOOLEXPR means that the boolean expression BOOLEXPR is assumed to be always true during the execution of the program.Any variable or expression is considered to represent the sequence of values it takes during the whole execution of the system and Lustre operators operate globally over these sequences. For example, VAR=EXPR means that the sequences of values associated with VAR and EXPR are identical.Expressions are made of variable identifiers, constants, usual arithmetic, boolean and conditional operators and two specific temporal operators: previous pre and followed-by →.• If E is an expression denoting the sequence (e_0, e_1,e_2,...), then pre(E) denotes the sequence (nil, e_0, e_1,...) where nil is an undefined value.• If E and F are two expressions of the same type respectively denoting the sequences (e_0, e_1, e_2,...) and (f_0,f_1, f_2,...), then E → F is an expression denoting the sequence (e_0, f_1, f_2,...).An example of specification is given below:This SCADE specification computes an output z6 (boolean) from five inputs: x (real), p1 to p4 (booleans). z1 is true if x is greater or equal to 2, z2 is z1 or p1, z3 is true if z2 is confirmed during at least 8 cycles, z4 is z3 or p2, z5 is the result of a latch, finally z6 is z5 or p4.xp2p1p3p44. Approach for automatic test generationThis section presents our objectives with respect to the current practices described in section 2, explains the proposed approach and briefly describes the structural coverage criterion we have chosen.ObjectivesThe main objective of the project was to study the feasibility of test generation automation based on formal techniques. It consisted in determining whether automatic test generation techniques and the associated tools were applicable to specifications of real industrial problems, but also in seeing if these techniques could automatically identify equivalence classes and singular points, two central notions used in current practice.A second issue was to evaluate the structural coverage obtained by a set of test cases. This is possible only if a pertinent structural coverage criterion is defined at the specification level.Test objectives and test casesBefore explaining the approach, we will define on the example of section 2 what a test objective and a test case are.Formally, a test objective is any property that can be expressed in SCADE. From a user point of view, different levels of test objectives are possible. The objective for the specification of section 3 was to have z6 true, but more detailed objectives are possible, they usually express relationships between inputs and outputs.A test case is a sequence of n steps - each step gives the value of the inputs and outputs of the considered system - such that the test objective is attained. One possible test case for the example specification of section 2 and the test objective (z6 = true) is the following:#CYCLE x p1 p2 p3 p4 z61 1false false true false false2 1true false true false false3 1true false true false false4 1true false true false false5 1true false true false false6 3 false false true false false7 4 false false true false false8 12 false false true false false9 6true false true false false10 5 false false false false trueThis test case is 10 cycles long. Each line represents one time cycle and gives the values of inputs and output at this cycle.ApproachFor a given system, a set of Functional Test Objectives (FTO) are defined. The chosen tools (TCG and GATEL, see section 5) are able to generate test cases from a SCADE specification of the system, a description of the environment (as SCADE assertions) and FTOs.At first, only one test case is defined for each FTO (step 1 in the following figure), but this is not sufficient. The behaviour class defined by the FTO has to be refined to generate several test cases for each FTO (step 2). A structural coverage criterion is then used to decide whether a sufficient refinement level has been obtained.Structural coverage criterionWe have chosen a structural coverage criterion at the specification level (adapted to SCADE specifications). The criterion has been defined by LSR-IMAG, for more information, see [LPB04].We do not give here the detailed definition of this criterion but try to describe its objective in terms of coverage:for each node and for each output variable of this node, all possible values of each input variable must be exercised in a context where its value has an influence on the value of the output variable.This criterion is very close to the constraints imposed by MCDC (Modified Condition Decision Coverage) [HVCR01], the main difference is that MCDC is defined at the code level while we are interested in a criterion at the specification level. To precisely compare both criteria, it would be necessary to study the transformations introduced by automatic code generation.Practically, we give for each basic SCADE operator a set of boolean conditions that must be set to true at least once by a test case for the operator to be covered. For example, an AND operator (o = AND(i1,i2)) is covered if the following boolean conditions are set to true at least once:not i1not i2 and i1i1 and i2.Within Airbus, a library of special nodes called symbols exists, symbols are skill oriented nodes that are very often used. Typical symbols are: filters, triggers, integrators. We also defined coverage conditions for these symbols, either by combining the conditions of the basic operators defining the symbol or by using designer expertise. For example, the conditions defined for a latch (o=latch(set,reset)) are the following :TC : Test Caseresetset and not resetnot reset and not set and pre(o)not reset and not set and not pre(o).Finally, time plays an essential role in the systems we consider; the criterion defines conditions to be covered at a given time cycle, but two issues must be answered to adequately handle time: we have first to be able to know how many cycles need tobe considered, and if n cycles are considered, we have to decide whether it is necessary to cover every boolean condition at every cycle. These two points havenot yet been definitely answered for the moment. We have explored different solutions depending on the tool (cf section 5).5. Test generation toolsWe have experimented two tools: GATEL and TCG. GATEL [MA00] is a tool developed by CEA (French Nuclear Research Centre) based on constraint resolution techniques. TCG (Test Case Generator) is a tool developed by Prover Technology () based on a proof kernel.GATELThe user provides a test description, i.e. a description of the environment, a description of the program under test and a test objective. The program is written in Lustre (textual formal specification language on which SCADE is based [CHPP87]), but GATEL only accepts booleans and integers (no real numbers, no structured types). The environment description is a set of hypotheses on the behaviour of the environment, it defines invariant relationships between the program inputs and the outputs at the previous instant. The goal of these hypotheses is to restrict the possible values of inputs to realistic ones with respect to the environment of the system. The test objective defines a property that must be satisfied by the generated test cases. This property may be an invariant (written in Lustre) that must be verified at each instant or a property that must be verified at least at one instant (additional operator reach).GATEL automatically generates a test case from this test description. Several test cases can also be obtained by interactively “splitting“ the operators included in the test description - it corresponds to a partition of the system into equivalence classes - and to generate one test case for each class. GATEL proposes default splitting rules for all operators, but a different strategy can also be encoded in the tool.An interesting functionality of GATEL is that it first produces test sequences: a test sequence is a test case where only the values that have an influence on the result are given, and where value intervals are given for numerical variables. For example, the test sequence corresponding to the test case given in section 4 is the following:#CYCLE x p1 p2 p3 p4 z61 __ _ _ _ _2 _true _ _ _ _3 _true _ _ _ _4 _true _ _ _ _5 _true _ _ _ _6 3..? _ _ _ _ _7 3..? _ _ _ _ _8 3..? _ _ _ _ _9 _true _ _ _ _10 3..? _ _ false _ trueA test sequence represents a class of test cases and gives interesting information on the system. Each test sequence can then be completely instantiated to produce a test case.GATEL kernel is a constraint solving algorithm on boolean variables and integer intervals. From the environment assertion, test objective and definition of the input and output variables, the tool creates a constraint system characterising the final state that must be reached by the lustre program. The resolution of this constraints system consists in progressively eliminating constraints introduced by the user in the test description. It is implemented by alternate steps of constraints propagation/simplification and variables instantiation. The resolution starts from the final state of the test case (computed from the test description) and incrementally builds previous states using a backward exploration technique.To apply GATEL to the example of section 3, we provide a textual version of the example and adapted splitting rules, then we define the test objective (reach z6). The tool first automatically produces one simple test sequence satisfying the objective:#CYCLE x p1 p2 p3 p4 z63 -56807 true false true false false2 121403 false true false true true1 -690771 true false false false true0 -774527 true false false false trueThen the user can refine this by interactively applying defined splitting rules. First, the rule for the OR operator defining z6 is applied, it results in two test sequences (one where z5 is true and one where p4 is true and z5 false). For each of these two possibilities, the user can further refine the test sequences by applying the splitting rules for BASCR. A test tree is progressively built by applying splitting rules from outputs to inputs. As an illustration, we give the test tree obtained by splitting the OR operator as explained above and the BASCR operator only in the branch where p4 is true (right branch).At this stage, we have only refined the test sequences at one time cycle (cycle 0). When all the splitting is done at one cycle, we can further refine by considering the next time cycle. On the example here, it is necessary to go at least till cycle 9 to see the effect of the CONF operator. The remaining question is whether to consider all the possible boolean combinations at each of the 9 cycles. GATEL allows all the combinations to be considered, but we quickly obtained a huge test tree that is difficult to handle. We think such an extensive splitting is not always necessary and we have defined a methodology to guide the user (a list of cases where no further splitting is needed). This kind of methodology could be adapted with respect to application domain and test habits.TCGThe user defines Functional Test Objectives (FTO) using SCADE. The tool also considers Structural Test Objectives (STO) which correspond by default to a branch coverage (one case for each branch of each conditional statement) but can be adapted to handle other coverage criteria (like the one we define in section 4). The tool runs in three steps.• First it generates test cases for the given functional test objectives while trying at the same time to cover as many structural test objectives as possible. The structural test objectives that are considered at this stage are only those that are in the dynamic cone of influence of the functional test objective (i.e. the set of variables that have an influence on the value of the FTO).• Then TCG can generate test cases for the remaining STO (these STO enlighten parts of the specification that are not exercised by the FTO).• Last, if there are still STO that have not been covered, TCG uses its proof capabilities to see whether these STO are reachable or not (it can help in identifying dead code).The main principle used in TCG for the test case generation is the finding of counter-examples. The tool searches for counter examples of the negation of the test objective in order to obtain scenarios where the test objective is satisfied.TCG is completely automated. The syntax of the produced test cases is compatible with the SCADE simulator. TCG is not yet completely integrated into SCADE but a prototype has been implemented to allow its use from the SCADE interface. TCG also produces an excel file that synthesizes the coverage of the objectives by the test cases. An example of this file is given below:namefunctional scenarios structural scenarios in symbol variablecov 0 1 2 3 0 1 2 3 4 cone s_1 por OR1 X X X X s_25OR2X s_25 OR3 X X s_25s_5 pand AND1X s_25 AND2 X X X X s_25AND3X s_25 s_13 pCONF1 CONF11 S X X s_25CONF12 X X X X s_25CONF13 S X X X s_25CONF14 S X X s_25s_18 pand AND1 S X X X X s_25AND2 X X X X s_25AND3 S X s_25s_22 pand AND1 S X X X X X s_25AND2 X X X X s_25AND3 S X s_25s_25 BRUDSVX X X XOn each line are the test objectives, first the structural test objectives for the operators (AND, OR, CONF1) then the functional test objective (here only one: BRUDSV). The columns give the test cases (or scenarios), there are 4 functional test cases (i.e. test cases generated during the first step) and 5 structural test cases (generated during the second step). A mark means that the test objective of this line is covered by the test scenario of this column. The ‘S’ denotes the test objectives that are only covered by structural scenarios.As far as time is concerned, TCG computes test cases of a given length (at least the number of cycles that is needed to reach the functional objective) and tries to cover each structural objective only at one time cycle, so that finally all the boolean combinations will be covered by the set of test cases, but maybe at different instants.Experimentations / ComparisonBoth tools have been experimented on several examples taken from Airbus specifications. Two main systems were studied: the flight control secondary computer and the flight warning system. Both tools have been able to generate interesting test cases and their good performances allowed us to deal with real size examples. Moreover, the tools have been parameterised to take into account the coverage criterion we have defined.However, implementations of the test generation approach are different in the two tools. TCG is completely automatic and can be integrated into the SCADE environment. The optimisation of the number of test cases is interesting. But test objectives are covered only at one time cycle and there is no notion of equivalence classes. GATEL provides the equivalence class notion that we consider essential and allows the coverage of test objectives at each time cycle, but it is more complex to use (interactive), the definition of a methodology is thus needed to guide the user. 6. Conclusion and future workThe main result of the project is to show that automatic test generation is indeed feasible in practice. Tools exist and are able to handle real industrial examples. An adapted criterion has been chosen to analyse coverage at the specification level. A few experimentations have been conducted to compare automatically generated tests with tests obtained with the current manual approach, but they are not sufficient. It would be interesting to fully compare both approaches by applying them in parallel on a given system with functional test objectives. As far as tools are concerned, methods proposed by GATEL and TCG could be combined to take advantage of their complementarity. More work is also necessary to analyse the cases where the tools could not generate a test case covering a given test objective. Finally, at the methodological level, the proposed approach is compliant with the certification standards. However, the proposed approach is different from the classical coverage analysis at the code level, so we think it is necessary to study more thoroughly the possible substitution of the coverage analysis at the code level by a coverage analysis at the specification level. This reflection should be part of a more global study of the integration of this approach to the actual verification and validation process.References[CHPP87] P. Caspi, N. Halbwachs, D. Pilaud, and J. Plaice. LUSTRE, adeclarative language for programming synchronous systems. In 14thSymposium on Principles of Programming Languages (POPL 87),Munich, pages 178-188. ACM Press, 1987.[HVCR01] K.J. Hayhurst, D.S. Veerhusen, J.J. Chilenski, L.K. Rierson. A practical tutorial on Modified Condition / Decision Coverage. NASA / TM-2001-210876.[LPB04] A. Lakehal, I. Parissis, L. du Bousquet. Critères de couverture structurelle de programmes Lustre. Proceedings of AFADL 2004. . [LMW01] O. Laurent, P. Michel, V. Wiels. Using formal verification techniques to reduce simulation and test effort. FME 2001. LNCS 2021, Jose NunoOliveira and Pamela Zave eds. Springer Verlag.[MA00] B. Marre, A. Arnould. Test Sequence Generation from Lustre Descriptions: GATEL. 15th international conference on AutomatedSoftware Engineering (ASE) 2000.。