The simplyTestify general architecture is sketched in the diagram above. The user (developer, tester, ops engineer):

  • deploys the services architecture under test by installing, configuring, and setting up its components wherever she decides to locate them: on-premises, on private clouds, and on public clouds;
  • drafts a few declarative artifacts (XML infosets) describing the services architecture topology, the test harness configuration plan, the behavior models, and the test policies;
  • implements the SUT Initialization API with initialization procedures for the components of the system under test (SUT);
  • calls, through the GUI and the Test Generate & Run API, the Test Automation Methods of the simplyTestify Portfolio. 

The simplyTestify software robots execute the invoked methods, i.e. generate synthetic test suites, run dynamically scheduled test sessions, reactively organize multi-session test campaigns with the on-the-fly synthesis of new test cases. Equipped with artificial intelligence and model checking technologies, they perform the effective and efficient search, detection and diagnosis of failures and provide keen reports to the users.

Why Test Automation?

Test automation is a buzzword: the tools currently available in the QA & Testing market implement at most the bare mechanization of some clerical tasks, such as the sending of the test stimuli to the system under test and the collection of its responses. 

Even with the support of these tools, customers must provide a massive effort of custom, tailor-made, and ad hoc software development to achieve a small level of test mechanization, always limited to the most clerical tasks. The resulting home-made test systems are brittle, non-optimized, and error prone, and their maintenance proves to be very complicated and challenging, often more expensive than the systems to be tested.

The design and generation of synthetic test cases is a hot topic. Everybody knows that the widespread use, for testing, of real data coming from the production systems is inefficient and ineffective because the test cases thus fabricated exhibit both reduced fault detection potential and limited coverage. 

Moreover, the recent EU Directive on Data Protection (Regulation 2016/679 of the European Parliament and of the Council of 27 April 2016) practically makes it impossible to use, for testing, "real data," even masked, unless not sensitive at all. 

That said, the hand-operated design and generation of synthetic test cases are knowledge intensive, low productivity, time-consuming, and error-prone (false positives, false negatives – the tester’s nightmare). 

The question for businesses is not whether but when – in any case, at most in two years from the Directive publication date - they will be able to regularly produce adequate amounts of sound, effective, and efficient synthetic test cases.

The configuration and deployment of the test harness and its binding to the distributed test bed is another critical task, accomplished by complicated, custom-built, and tailor-made software whose development and maintenance require high-level technical skills. 

The tester is obliged to implement not only a testbed but also an ad hoc test harness for each component under test, and to adapt it to the business and technological evolutions of both the component to be tested and the equipment it runs on.

Cognitive test run tasks, such as: 

  • the intelligent dynamic prioritization of the tests (which is a must for testing efficiency), 
  • the evaluation of the SUT response (technically speaking, their arbitration, i.e. the production of test verdicts), and
  • the drafting of the test reports from bulky journals, 

are nowadays carried out manually.

The organization of test campaigns is a harsh and complicated endeavor. Businesses and Administrations put in place human workflows, eventually with the help of a workflow management tool, for the control of test campaigns and their integration within the DevOps processes. 

The decision points in these workflows are challenged with almost unsolvable problems such as determining when starting and stopping a testing phase, which runs on disputed equipment, is carried out by scarce human resources, and is supported by unstable test systems, upon software whose quality continues to be perceived as uncertain.

Moreover, test automation is henceforth a necessity. Beyond a very low level of complexity and scale, already surpassed by the contemporary distributed systems, and even without considering new ones including IoT-enabled services and connected objects, the implementation of adequate manual testing practices requires the sustained and simultaneous mobilization of:

  • costly business, technical, and testing skills, which are difficult to find and organize, as well as
  • constant attention and observation capabilities, well beyond the capacities of human beings.

In summary, present and future test challenges generate increasing needs that are not satisfied by the currently available technology and can be met only by full test automation. 

Testing functionalities and patterns

The simplyTestify technology targets black-box test of individual components and monolithic applications. It proposes the automated design and production of synthetic test suites, i.e. collection of test scenario samples made of canned stimuli (requests, replies, and one-way messages) and oracles (representations of expected SUT responses - requests, replies, and one-way messages - of the distributed SUT). 

It also automatically deploys a test harness that incorporates the simulation of upstream and downstream service dependencies of the distributed SUT: it automatically creates test components (clients, stubs) able to send the canned stimuli to the system under test and to gather and compare its responses with the oracles.

The simplyTestify technology enables client-driven test of third party services, supplied by other departments and external vendors. 

Within client-driven testing, the user models the third party service behavior that her components expect. As usual, simplyTestify robots generate the test suites, deploy the test harnesses with the appropriate test components, dynamically prioritize, run, arbitrate the tests, and return the test reports and journals. 

A failure verdict on a third party service response reveals a mismatch between the expected and the actual behavior of the service from the client point of view. In this case, either the service functions and behaviors do not fulfill the client's requirements, as set out, correctly or incorrectly, in the SUT models, or the implementation of the service is faulty. The precocious discovery of both kinds of failures is very useful, from the pre-procurement and evaluation stages, up to the incremental integration of the service in the user's architecture.

The simplyTestify technology enables gray-box test of distributed SUTs that can be large scale, and cross-organizational. In gray-box testing, the SUT components are perceived as black-boxes, but the interactions between them through the service dependency wires can be observed and evaluated. 

The generator robot can identify all the end-to-end interaction scenarios in a distributed SUT, and design and generate an optimized number of effective and efficient synthetic test samples, i.e. instances of these scenarios made of canned stimuli and oracles.  

The executor/arbiter robot creates and places, in addition to the modeled clients and stubs, the intercepting proxies on the service dependency wires designated to be monitored in the Test Harness Configuration Plan, and can manage the execution of asynchronous, non-deterministic test scenarios, where some SUT responses can arrive in an unspecified order. 

The scheduler/planner robot can prioritize dynamically the test samples in a session, ad plans a multi-session test campaign by the use of probabilistic reasoning, under the guidance of the user-defined policies.

Full, autonomic, intelligence-led, robotics, and low-code Test Automation

simplyTestify carries out autonomic, intelligence-led, robotics and low-code Test Automation as a Service, through the combination of robotization, optimization, and routinization of all the test tasks.

Robotization

The simplyTestify Test Automation Methods are executed by software robots that perform autonomously the test tasks humans assign. Once configured by models and guided by policies, the robot can be left to accomplish the work by itself, and can interact with the system under test the same way a human does. The robots' autonomous activities carry out not only the clerical but also the cognitive test tasks, such as:

  • the intelligent design and generation of synthetic test stimuli, 
  • the generation of oracles by the SUT topology and behavioral models,
  • the test harness configuration and deployment with the model-driven and automated creation of simulated upstream/downstream components (clients/stubs) and of intercepting proxies on the service dependency wires;
  • the binding of the test harness to the multi-component SUT deployed on distributed testbeds,
  • the intelligent dynamic prioritization of the test samples; 
  • the arbitration of the system under test responses; 
  • the drafting of the test session reports;
  • the evidence-based, reactive organization of multi-session test campaigns.

A diagram of the core robot architecture is sketched in the figure below.

Optimization 

simplyTestify robots employ advanced and efficient techniques of model checking, temporal logic inference, and probabilistic graphical reasoning with Bayesian Networks (compiled in Arithmetic Circuits to increase the performance), to optimize the search, detection, and diagnosis of failures performed by the software robots. These techniques enable:

  • the model-driven production of optimized synthetic test suites, based on model checking and temporal logic of action inference, by the generator robot,
  • the dynamic, context-aware prioritization of the test samples based on probabilistic inference by the scheduler robot, and
  • the reactive, evidence-based planning of test campaigns based on the ongoing trace of the test runs and responses, with the on-the-fly generation of new test samples, by the planner robot.

Routinization

simplyTestify is a reliable programmable service. It allows:

  • the reliable invocation of Test Automation Methods via the Test Generation & Run API, which is tolerant to transient network faults, and
  • their trusted execution by autonomic robots – equipped with auto scaling and self-recovery mechanisms - on a highly scalable, available, serviceable, secure and performant infrastructure.

In this way, the user's DevOps environments, processes, and managers can safely invoke the simplyTestify Test Automation Methods, enabling the smooth and reliable wrapping of automated:

  • black-box test of single services starting from the early stages of design (progression testing, shift left testing);
  • black-box test of partners’ and third-parties’ components (client-driven testing), starting from the initial phases of the provisioning process (pre-procurement, evaluation testing);
  • gray-box test of distributed systems throughout the process of incremental integration (integration testing);
  • test of upgraded component builds (change testing, regression testing);
  • on-the-fly test of suspicious components by the ops monitoring system (shift right testing).

To sum up, simplyTestify makes it possible to put Test Automation at its real place, as a pivot activity invokable in all the DevOps phases (see the figure below), and to support Agile, Test Driven and Behavior Driven Development methodologies, and Continuous Integration and Delivery DevOps processes.

Model and policy based, low-code and intelligent Test Automation

The simplyTestify robots, to deliver full Test Automation, take into account interfaces and protocols of the distributed system under test in their native description languages (e.g. WSDL for SOAP, WADL or WSDL for REST/XML, Open API/Swagger for REST/JSON ...).

Furthermore, the user is requested to draft a few intuitive, compact, and declarative artifacts in XML format, representing:

  • the Distributed SUT Topology, namely the graph of components linked by service dependency wires;
  • the Test Harness Configuration Plan, through the simple indication of (i) the components to be simulated (clients and stubs), and (ii) the service dependency wires to be observed (intercepting proxies);
  • for each actual or simulated component, the Protocol State Machine that describes the conversation with the upstream and downstream service components, namely the causal (state transitions) and data flow (transfer functions) associations between ingoing and outgoing messages;
  • the Generation, Scheduling, and Planning Policies.

The only coding effort requested from the user concerns the initialization procedure for each component under test. These procedures implement the simplyTestify SUT Initialization API, which is automatically invoked by the simplyTestify executor/arbiter robot on all the SUT components. 

The SUT Initialization API enables the executor/arbiter to meet the general requirement that each test run must leave the system under test in its initial state, letting it ensure the repeatability of the test runs. 

Furthermore, simplyTestify enables contract-based testing. The user is allowed to model pre/post-conditions on stateful components. She shall define a state resource (XML infoset) that represents the externally visible state of the component (in fact, the view of its state that is meaningful for its clients) and implement the Contract-Based Test API for setting and getting this resource. 

The executor/arbiter robot automatically invokes this API on the SUT components at the appropriate moments of the test cycle, to set the initial state at the beginning of the test run and to get the actual state whenever it is necessary to check whether it meets the pre/post-conditions.

Users guide and focus the test generation process with Test Generation Policies. For instance, the user can concentrate the synthesis of test inputs on particular message types - to enable the exploration of specific logical and “physical” regions of the distributed SUT - and focus on specific ranges of business data values to fill the inputs' contents. 

At each test run, the scheduler's probabilistic inference engine recomputes by the evidence of the SUT behavior (the past test verdicts) the failure discovery potential, namely the probability of failing, of the remaining test samples. 

The test scheduling policies can influence the dynamic prioritization of the test samples, i.e. the choice of the next test sample to run. In principle, the Max Potential Policy ("the most fallible the first") increases the fault detection rate and implements a depth-first search for failure. 

Otherwise, the Max Coverage Policy  ("the least informed the first") dynamically sorts the test samples by their Shannon entropy - the max entropy being the probability distribution of the test sample Boolean variable (pass/fail) that is the nearest to fifty-fifty. The Max Coverage Policy drives a breadth-first search of faulty elements in the distributed SUT.

The Shannon entropy of the SUT functional decomposition elements is used by the planner robot to dynamically formulate test generation policies that recommend the synthesis of new test samples involving scarcely tested or untested message types, interfaces, and SUT components. 

The Planning Policy establishes whether and when the test session manager must stop the current session and invoke the test generator robot with the appropriate Test Generation Policy.

Note that the drafting of the models and policies listed above and the implementation of the SUT Initialization and the Contract-Based Test APIs require only knowledge of the topology and of the external behavior of the distributed SUT. In other terms, the user builds an external model of the SUT as it should be and behave, without employing any particular testing skill.

Compared to competitor QA & Testing tools, the simplyTestify Full Test Automation technology, by the intelligent automation of the cognitive test tasks, increases the efficiency and efficacy of search, detection, and diagnosis of failures by one order of magnitude. 

Full Test Automation on Demand

simplyTestify is a Test Automation Platform as a Service that proposes pay-per-use and low-cost self-provisioning, without any investment in licenses, equipment, and custom-built, high code test systems. It is easy to try, understand, learn, and use.

Each user instance of the simplyTestify test harness runs in a sandbox and interacts securely with the SUT components wherever they are, on the Internet or via a direct connection. 

Compared to competitor QA & Testing tools, simplyTestify lowers the cost of testing distributed services architectures by one order of magnitude. All the competitors propose expensive licensing approaches, either on-premises or on cloud, which require additional substantial human and equipment investments. The result is the bare mechanization of some clerical testing task, which nevertheless requires expensive and complex ad hoc software developments (high-code automation).   

Test Method Platform as a Service 

simplyTestify implements a Test Method Platform as a Service that offers to developers, researchers, and practitioners the environment, the mechanisms, the guidelines, and the support for importing, installing, configuring, and deploying their advanced testing tools as Test Automation Methods on the Platform. These imported Methods, when analyzed, evaluated, tested, and certified by Simple Engineering, can be inserted in the simplyTestify Test Automation Method Portfolio. 

Once included in the simplyTestify Portfolio, there is no difference between the imported and the in-house Methods. The Platform proposes all the Methods in the same manner: they are listed in the Portfolio, can be invoked via the GUI and the Test Generation & Run API, and run on the elastic and reliable simplyTestify Platform. Per-use royalties remunerate the imported Method developers.

Conclusion

In an always-on world, the enterprise’s most vital applications and services become mission-critical. Each component in the chain of a mission-critical application is essential, from the mobile device to the back-end databases and the downstream third-party services. If any link is faulty and the application does not work, it does not matter to the end user which piece fails, and the failure can impact not only revenues and costs but also the company’s reputation.

Preventing defects from reaching production and fixing them immediately when discovered are the essential aims and the promises of effective and efficient test automation. Businesses can catch the highest chance to attain this objective with the autonomic, intelligence-led, robotics, and low code, extreme, and intelligent test automation on demand offered by simplyTestify.

For additional information, download simplyTestify - The Test Automation Platform as a Service     

To join the simplyTestify Early Adopter Program, please contact

Libero MAESANO

libero.maesano@simplytestify.com