.. _testing_knowledge: ================= Testing Knowledge ================= General Approach and Guidelines -------------------------------- In general, integration tests are conducted using the following strategy: first of all, functional tests are defined using the FSMT framework. Secondly, these tests are added to the **CSRA Test Distribution** as mentioned in the Tooling section. Thirdly, so-called "run" jobs must be defined in order to automatically execute the previously defined tests on a CI Server, please also see Tooling Section: **CSRA RUN Tests Jobs**. Functional tests can "assess" basal properties such as, is the sub-system starting correctly on localhost, the CI Server in this case. More advanced tests however, may test if the apartment currently provides the correct/required information/sensory data. Therefore, you may also interact with the sensors (hardware) in the apartment by connecting to it (from the CI Server). In these advanced tests you may evaluate "run time" data based on the apartment's sensory output. See Workflow section. Required Tools -------------- 1. `FSMT Testing Framework `_ (Already included in the CSRA-nightly Distribution) .. code-block:: bash git clone https://opensource.cit-ec.de/git/fsmt && cd fsmt && git checkout 0.18 2. CSRA `Integration Tests Distribution `_ .. code-block:: bash git clone https://opensource.cit-ec.de/git/citk 3. CSRA Test Git Repository (Already included in the Integration Tests Distribution) .. code-block:: bash git clone https://projects.cit-ec.uni-bielefeld.de/git/lsp-csra.fsmt-experiments.git 4. CSRA RUN Tests Jobs (Already included in the Integration Tests Distribution, see: **runnable** jobs) .. code-block:: bash git clone https://opensource.cit-ec.de/git/citk Testing Concept Overview ------------------------- .. figure:: /images/Testing-Concept-CSRA.png :figclass: align-center Basically, the CSRA Testing Distribution contains two kinds of recipes: a) FSMT test definitions, and b) "runnables". As soon as the build-generator finishes, the aforementioned build job types are available on the CI Server. While test definitions are installed using CMake (static), runnables are, in contrast, CITK recipes that include a shell fragment which actually invokes FSMT with a given test definition (that has just been installed by CMake), such as: .. code-block:: bash #!/bin/bash sleep 10; export DISPLAY=:0.0; . ${toolkit.dir}/etc/vdemo_scripts/lsp-csra-base.sh; $prefix/bin/fsmt -o $WORKSPACE -c ${toolkit.dir}/etc/fsmt-experiments/mixed/audio/audio-io-test-hallway.scxml;" FSMT tests jobs (runnables) are executed on a dedicated **build slave** (mex) attached to the CSRA CI server (ayq). This build slave is part of the apartment's network (spread ring) and may access all data sources/sensors/files that are provided by the apartment. This way tests may either test locally, that is, on the build slave with a local spread configuration for instance, or connect globally to any core machine or service. The tests results are saved in the CI Server's workspace (per job) and are accesible by all devs. Current Testing Infrastructure ------------------------------- The current testing infrastucture comprises a) our `Jenkins CI server `_, b) a dedicated **testing build slave**, and c) the apartment, of course. The testing distribution is generated by executing this `build job `_ In case the tests changed, you need to update/install them by executing `this job `_ After that, you may trigger any job by executig one of the **runnable** jobs. Currently, all integration tests are automatically `triggered every morning `_ at 3:15 AM, sending an email report to the CSRA-DEV mailing list. Test Integration ----------------- TODO: Describe physical "Trigger Test" Button in Apartment TODO: Describe Testing State Machine with Speech output Test Templates --------------- Please place this information on top of everything else in your ``.ini`` file. .. note:: New lines require a **leading** _whitespace_ .. code-block:: bash [meta] description = "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no # sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea # rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet." input_data = "/rsb/scope/x/y, file_xy, /dev/urandom" expected_result = "x is greater than y, time is lower than, message of type x was received ... Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua." Example Test INI ----------------- `Example Ini `_ Overview Application -------------------- We also have a "constant" visual reminder of how many test are currently passing without actually opening a web-browser and looking at the CI Server. This application is meant to provide a quick overview. .. figure:: /images/CSRA-Codesprint-7.jpg :figclass: align-center Ground Truth Data ------------------ If you previously recorded ground truth data that you might test against in your integration test, files *larger than 2MB* are to be saved in... .. code-block:: bash $prefix/share/fsmt-experiments ...where ``$prefix`` is the current distribution, e.g., ``/vol/csra/releases/trusty/lsp-csra-nightly/`` .. note:: Files **<= 2MB** may be checked in with your ini files. Current Tests -------------- These are the currently implemented integration tests: Flobi ^^^^^ `Flobi Simulation `_ `Flobi HLC Test `_ Audio ^^^^^ Homeautomation ^^^^^^^^^^^^^^^ Middleware ^^^^^^^^^^^ Speech ^^^^^^^ Situation ^^^^^^^^^^ Infrastructure ^^^^^^^^^^^^^^^ Memory ^^^^^^^ System Status ^^^^^^^^^^^^^ https://projects.cit-ec.uni-bielefeld.de/projects/lsp-csra/repository/fsmt-experiments/revisions/master/entry/physical/components-via-vdemo.ini