We use Travis CI for our continuous integration server. It automatically runs our battery of thousands of unit/integration/screenshot tests after each commit to our GIT repo. More information at the links:
- Matomo on Travis CI: https://travis-ci.com/matomo-org/matomo
Each core Matomo developer is responsible to keep the build green. If a developer breaks the build, they will receive an E-Mail from Travis CI.
The next section explains how you can run the test suite on your own dev machine.
How To Run Matomo Tests
To run tests, you must install Matomo via Git and set it up for development. A guide for this is available in our Developer Zone. The part about "Creating a plugin" can be skipped.
To execute the tests:
In your php.ini make sure you have the setting to show all errors:
error_reporting = E_ALL | E_STRICT
Go to tests/index.html to see the tests homepage and run the Integration tests via a visual UI, or run JS Tests
Next you will need to install PHPUnit
To install PHPUnit, run
php composer.phar install --devin the Matomo root directory.
matomo/config/config.ini.phpis set up correctly, i.e. with the correct password to prevent the following error:
SQLSTATE  Access denied for user 'root'@'localhost' (using password: NO)
Run the tests
$ cd /path/to/matomo $ ./console tests:run --testsuite unit $ ./console tests:run --testsuite integration $ ./console tests:run --testsuite system
There are also two main groups of tests: core and plugins For example run
./console tests:run coreto run all Core Matomo tests.
You can combine testsuite and groups like this:
./console tests:run --testsuite unit core. This would run all unit tests in core.
./console tests:run --testsuite integration CustomAlerts. This would run all integration tests of the CustomAlerts plugin.
./console tests:run CustomAlerts. This would run all unit, integration and system tests of the CustomAlerts plugin. (group only)
To execute multiple groups you can separate them via a comma:
./console tests:run CustomAlerts,Insights. This would run all unit, integration and system tests of the CustomAlerts and Insights plugin.
Write more tests :) See "Writing Unit tests with PHPUnit"
How to differentiate between unit, integration or system tests?
This can be sometimes hard to decide and often leads to discussions. We consider a test as a unit test when it tests only a single method or class. Sometimes two or three classes can still be considered as a Unit for instance if you have to pass a dummy class or something similar but it should actually only test one class or method. If it has a dependency to the filesystem, web, config, database or to other plugins it is not a unit test but an integration test. If the test is slow it is most likely not a unit test but an integration test as well. "Slow" is of course very subjective and also depends on the server but if your test does not have any dependencies your test will be really fast.
It is an integration test if you have any dependency to a loaded plugin, to the filesystem, web, config, database or something similar. It is an integration test if you test multiple classes in one test.
It is a system test if you - for instance - make a call to Matomo itself via HTTP or CLI and the whole system is being tested.
Why do we split tests in unit, integration, system and ui folders?
Because they fail for different reasons and the duration of the test execution is different. This allows us to execute all unit tests and get a result very quick. Unit tests should not fail on different systems and just run everywhere for example no matter whether you are using NFS or not. Once the unit tests are green one would usually execute all integration tests to see whether the next stage works. They take a bit longer as they have dependencies to the database and filesystem. The system and ui tests take the most time to run as they always run through the whole code.
Another advantage of running the tests separately is that we are getting a more accurate code coverage. For instance when running the unit tests we will get the true code coverage as they always only test one class or method. Integration tests usually run through a lot of code but often actually only one method is supposed to be tested. Although many methods are not tested they would be still marked as tested when running integration tests.
System tests files are in
System tests allow to test how major Matomo components interact together. A test will typically generate hits to the Tracker (record visits and page views) and then test all API responses and for each API output. It then checks that they match expected XML (or CSV, json, etc.). If a test fails, you can compare the processed/ and expected/ directories in a graphical text compare tool, such as WinMerge on Win, or MELD on Linux, or even with PhpStorm, to easily view changes between files.
For example using Meld, click on "Start new comparison", "Directory comparison", in "Original" select "path/to/matomo/tests/PHPUnit/System/expected" in "Mine" select "path/to/matomo/tests/PHPUnit/System/processed"
If changes are expected due to the code changes you make, simply copy the file from processed/ to expected/, and test will then pass. Copying files is done easily using Meld (ALT+LEFT). Otherwise, if you didn't expect to modify the API outputs, it might be that your changes are breaking some features unexpectedly.
Fixtures for System tests
System tests use Fixtures to generate controlled web usage data (visits, goals, pageviews, events, site searches, content tracking, custom variables, etc.).
Fixtures are stored in tests/PHPUnit/Fixtures
We also have an OmniFixture that includes all other Fixtures. OmniFixture is used for screenshot tests to provide data across most reports.
Keep OmniFixture up to date
Remember to update the Omnifixture SQL dump whenever you make any change to any fixture. You can use:
./console tests:setup-fixture OmniFixture --sqldump=OmniFixture-dump.sql
Keeping the OmniFixture up to date makes it easier to see which tests fail after each small fixture change.
If we don't update the OmniFixture then we end up with many failed screenshots tests which makes it hard to see whether those changes are expected or not.
Scheduled Reports Tests
As part of our system tests we generate the scheduled reports (in HTML, PDF & SMS).
Some of these scheduled reports contain PNG graphs. Depending on the system under test, generated images can differ.
Therefore, PNG graphs are only tested and compared against "expected" graphs, if the system under test has the same characteristics as the integration server.
The characteristics of the integration server are described in
Running tests on Ubuntu
If you use Ubuntu or another Linux distribution, you must make one change to the filesystem configuration to make tests run fast. Read more here.
See tests/README.testing-data.md to import testing data in Matomo.
UI Screenshots Tests
Download build artifacts for any recent commit
You can retrieve the files generated during the build (the build artifacts) at builds-artifacts.matomo.org
Test logs on CI
By default tests running on Travis CI will log all messages of at least
INFO level in
$PIWIK_ROOT_DIR/tmp/logs/matomo.log. In a given travis build output, you can view the logs by clicking on the line
$ cat $PIWIK_ROOT_DIR/tmp/logs/matomo.log at the end of the build output text.
VERBOSE messages are not logged by default (to keep Travis page loading fast). At any time you can temporarily enable logging by modifying this file and changing
log_level = info to
log_level = debug or
log_level = verbose.
Screenshot tests build artifacts
The screenshot tests generated by the continuous integration server are uploaded in builds-artifacts.matomo.org/ui-tests.master/
See tests/README.troubleshooting.md for troubleshooting the tests.
As a software developer writing tests it can be useful to be able to set breakpoints and debug while running tests. If you use Phpstorm read this answer to learn to configure Phpstorm with the PHPUnit from Composer.
See tests/PHPUnit/Benchmarks/README.md to learn about running Benchmark tests.
See tests/README.xhprof.md for help on how to profile Matomo with XHProf.
You can help by improving existing tests, or identify some missing tests and implement them. See https://matomo.org/participate/development-process Please contact us at firstname.lastname@example.org