Model-based testing
Model-based testing is an application of model-based design for designing and optionally also executing artifacts to perform software testing or system testing. Models can be used to represent the desired behavior of a system under test (SUT), or to represent testing strategies and a test environment. The picture on the right depicts the former approach.
A model describing a SUT is usually an abstract, partial presentation of the SUT's desired behavior. Test cases derived from such a model are functional tests on the same level of abstraction as the model. These test cases are collectively known as an
Tests can be derived from models in different ways. Because testing is usually experimental and based on heuristics, there is no known single best approach for test derivation. It is common to consolidate all test derivation related parameters into a package that is often known as "test requirements", "test purpose" or even "use case(s)". This package can contain information about those parts of a model that should be focused on, or the conditions for finishing testing (test stopping criteria).
Because test suites are derived from models and not from source code, model-based testing is usually seen as one form of black-box testing.
Model-based testing for complex software systems is still an evolving field.
Models
Especially in
Deploying model-based testing
There are various known ways to deploy model-based testing, which include online testing, offline generation of executable tests, and offline generation of manually deployable tests.[2]
Online testing means that a model-based testing tool connects directly to an SUT and tests it dynamically.
Offline generation of executable tests means that a model-based testing tool generates test cases as computer-readable assets that can be later run automatically; for example, a collection of Python classes that embodies the generated testing logic.
Offline generation of manually deployable tests means that a model-based testing tool generates test cases as human-readable assets that can later assist in manual testing; for instance, a PDF document in a human language describing the generated test steps.
Deriving tests algorithmically
The effectiveness of model-based testing is primarily due to the potential for automation it offers. If a model is machine-readable and formal to the extent that it has a well-defined behavioral interpretation, test cases can in principle be derived mechanically.
From finite state machines
Often the model is translated to or interpreted as a
Depending on the complexity of the system under test and the corresponding model the number of paths can be very large, because of the huge amount of possible configurations of the system. To find test cases that can cover an appropriate, but finite, number of paths, test criteria are needed to guide the selection. This technique was first proposed by Offutt and Abdurazik in the paper that started model-based testing.[3] Multiple techniques for test case generation have been developed and are surveyed by Rushby.[4] Test criteria are described in terms of general graphs in the testing textbook.[1]
Theorem proving
Constraint logic programming and symbolic execution
Constraint programming can be used to select test cases satisfying specific constraints by solving a set of constraints over a set of variables. The system is described by the means of constraints.[6] Solving the set of constraints can be done by Boolean solvers (e.g. SAT-solvers based on the Boolean satisfiability problem) or by numerical analysis, like the Gaussian elimination. A solution found by solving the set of constraints formulas can serve as a test cases for the corresponding system.
Constraint programming can be combined with symbolic execution. In this approach a system model is executed symbolically, i.e. collecting data constraints over different control paths, and then using the constraint programming method for solving the constraints and producing test cases.[7]
Model checking
Model checkers can also be used for test case generation.[8] Originally model checking was developed as a technique to check if a property of a specification is valid in a model. When used for testing, a model of the system under test, and a property to test is provided to the model checker. Within the procedure of proofing, if this property is valid in the model, the model checker detects witnesses and counterexamples. A witness is a path where the property is satisfied, whereas a counterexample is a path in the execution of the model where the property is violated. These paths can again be used as test cases.
Test case generation by using a Markov chain test model
See also
- Domain-specific language (DSL)
- Domain-specific modeling (DSM)
- Model-driven architecture (MDA)
- Model-driven engineering (MDE)
- Object-oriented analysis and design (OOAD)
- Time Partition Testing(TPT)
References
- ^ a b Paul Ammann and Jeff Offutt. Introduction to Software Testing, 2nd edition. Cambridge University Press, 2016.
- ISBN 978-0-12-372501-1, Morgan-Kaufmann 2007
- ^ Jeff Offutt and Aynur Abdurazik. Generating Tests from UML Specifications. Second International Conference on the Unified Modeling Language (UML ’99), pages 416-429, Fort Collins, CO, October 1999.
- ^ John Rushby. Automated Test Generation and Verified Software. Verified Software: Theories, Tools, Experiments: First IFIP TC 2/WG 2.3 Conference, VSTTE 2005, Zurich, Switzerland, October 10–13. pp. 161-172, Springer-Verlag
- S2CID 5774837.
- ^ Jefferson Offutt. Constraint-Based Automatic Test Data Generation. IEEE Transactions on Software Engineering, 17:900-910, 1991
- ^ Antti Huima. Implementing Conformiq Qtronic. Testing of Software and Communicating Systems, Lecture Notes in Computer Science, 2007, Volume 4581/2007, 1-12, DOI: 10.1007/978-3-540-73066-8_1
- ^ Gordon Fraser, Franz Wotawa, and Paul E. Ammann. Testing with model checkers: a survey. Software Testing, Verification and Reliability, 19(3):215–261, 2009. URL: [1]
- ^ Helene Le Guen. Validation d'un logiciel par le test statistique d'usage : de la modelisation de la decision à la livraison, 2005. URL:ftp://ftp.irisa.fr/techreports/theses/2005/leguen.pdf
- S2CID 9582606.
- ISBN 978-3843903486.
Further reading
- OMG UML 2 Testing Profile; [2]
- Bringmann, E.; Krämer, A. (2008). "2008 International Conference on Software Testing, Verification, and Validation". 2008 International Conference on Software Testing, Verification, and Validation. International Conference on Software Testing, Verification, and Validation (ICST). pp. 485–493. ISBN 978-0-7695-3127-4.
- Practical Model-Based Testing: A Tools Approach, Mark Utting and Bruno Legeard, ISBN 978-0-12-372501-1, Morgan-Kaufmann 2007.
- Model-Based Software Testing and Analysis with C#, Jonathan Jacky, Margus Veanes, Colin Campbell, and Wolfram Schulte, ISBN 978-0-521-68761-4, Cambridge University Press 2008.
- Model-Based Testing of Reactive Systems Advanced Lecture Series, LNCS 3472, Springer-Verlag, 2005. ISBN 978-3-540-26278-7.
- Hong Zhu; et al. (2008). AST '08: Proceedings of the 3rd International Workshop on Automation of Software Test. ACM Press. ISBN 978-1-60558-030-2.
- Santos-Neto, P.; Resende, R.; Pádua, C. (2007). "Proceedings of the 2007 ACM symposium on Applied computing - SAC '07". Proceedings of the 2007 ACM symposium on Applied computing - SAC '07. ISBN 978-1-59593-480-2.
- Roodenrijs, E. (Spring 2010). "Model-Based Testing Adds Value". Methods & Tools. 18 (1): 33–39. ISSN 1661-402X.
- A Systematic Review of Model Based Testing Tool Support, Muhammad Shafique, Yvan Labiche, Carleton University, Technical Report, May 2010.
- Zander, Justyna; Schieferdecker, Ina; ISBN 978-1-4398-1845-9.
- 2011/2012 Model-based Testing User Survey: Results and Analysis. Robert V. Binder. System Verification Associates, February 2012