unit_testing.rst 20 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457
  1. .. _doc_unit_testing:
  2. Unit testing
  3. ============
  4. Godot Engine allows to write unit tests directly in C++. The engine integrates
  5. the `doctest <https://github.com/onqtam/doctest>`_ unit testing framework which
  6. gives ability to write test suites and test cases next to production code, but
  7. since the tests in Godot go through a different ``main`` entry point, the tests
  8. reside in a dedicated ``tests/`` directory instead, which is located at the root
  9. of the engine source code.
  10. Platform and target support
  11. ---------------------------
  12. C++ unit tests can be run on Linux, macOS, and Windows operating systems.
  13. Tests can only be run with editor ``tools`` enabled, which means that export
  14. templates cannot be tested currently.
  15. Running tests
  16. -------------
  17. Before tests can be actually run, the engine must be compiled with the ``tests``
  18. build option enabled (and any other build option you typically use), as the
  19. tests are not compiled as part of the engine by default:
  20. .. code-block:: shell
  21. scons tests=yes
  22. Once the build is done, run the tests with a ``--test`` command-line option:
  23. .. code-block:: shell
  24. ./bin/<godot_binary> --test
  25. The test run can be configured with the various doctest-specific command-line
  26. options. To retrieve the full list of supported options, run the ``--test``
  27. command with the ``--help`` option:
  28. .. code-block:: shell
  29. ./bin/<godot_binary> --test --help
  30. Any other options and arguments after the ``--test`` command are treated as
  31. arguments for doctest.
  32. .. note::
  33. Tests are compiled automatically if you use the ``dev_mode=yes`` SCons option.
  34. ``dev_mode=yes`` is recommended if you plan on contributing to the engine
  35. development as it will automatically treat compilation warnings as errors.
  36. The continuous integration system will fail if any compilation warnings are
  37. detected, so you should strive to fix all warnings before opening a pull
  38. request.
  39. Filtering tests
  40. ~~~~~~~~~~~~~~~
  41. By default, all tests are run if you don't supply any extra arguments after the
  42. ``--test`` command. But if you're writing new tests or would like to see the
  43. successful assertions output coming from those tests for debugging purposes, you
  44. can run the tests of interest with the various filtering options provided by
  45. doctest.
  46. The wildcard syntax ``*`` is supported for matching any number of characters in
  47. test suites, test cases, and source file names:
  48. +--------------------+---------------+------------------------+
  49. | **Filter options** | **Shorthand** | **Examples** |
  50. +--------------------+---------------+------------------------+
  51. | ``--test-suite`` | ``-ts`` | ``-ts="*[GDScript]*"`` |
  52. +--------------------+---------------+------------------------+
  53. | ``--test-case`` | ``-tc`` | ``-tc="*[String]*"`` |
  54. +--------------------+---------------+------------------------+
  55. | ``--source-file`` | ``-sf`` | ``-sf="*test_color*"`` |
  56. +--------------------+---------------+------------------------+
  57. For instance, to run only the ``String`` unit tests, run:
  58. .. code-block:: shell
  59. ./bin/<godot_binary> --test --test-case="*[String]*"
  60. Successful assertions output can be enabled with the ``--success`` (``-s``)
  61. option, and can be combined with any combination of filtering options above,
  62. for instance:
  63. .. code-block:: shell
  64. ./bin/<godot_binary> --test --source-file="*test_color*" --success
  65. Specific tests can be skipped with corresponding ``-exclude`` options. As of
  66. now, some tests include random stress tests which take a while to execute. In
  67. order to skip those kind of tests, run the following command:
  68. .. code-block:: shell
  69. ./bin/<godot_binary> --test --test-case-exclude="*[Stress]*"
  70. Writing tests
  71. -------------
  72. Test suites represent C++ header files which must be included as part of the
  73. main test entry point in ``tests/test_main.cpp``. Most test suites are located
  74. directly under ``tests/`` directory.
  75. All header files are prefixed with ``test_``, and this is a naming convention
  76. which the Godot build system relies on to detect tests throughout the engine.
  77. Here's a minimal working test suite with a single test case written:
  78. .. code-block:: cpp
  79. #ifndef TEST_STRING_H
  80. #define TEST_STRING_H
  81. #include "tests/test_macros.h"
  82. namespace TestString {
  83. TEST_CASE("[String] Hello World!") {
  84. String hello = "Hello World!";
  85. CHECK(hello == "Hello World!");
  86. }
  87. } // namespace TestString
  88. #endif // TEST_STRING_H
  89. .. note::
  90. You can quickly generate new tests using the ``create_test.py`` script found in the ``tests/`` directory.
  91. This script automatically creates a new test file with the required boilerplate code in the appropriate location.
  92. It's also able to automatically include the new header in ``tests/test_main.cpp`` using invasive mode (``-i`` flag).
  93. To view usage instructions, run the script with the ``-h`` flag.
  94. The ``tests/test_macros.h`` header encapsulates everything which is needed for
  95. writing C++ unit tests in Godot. It includes doctest assertion and logging
  96. macros such as ``CHECK`` as seen above, and of course the definitions for
  97. writing test cases themselves.
  98. .. seealso::
  99. `tests/test_macros.h <https://github.com/godotengine/godot/blob/master/tests/test_macros.h>`_
  100. source code for currently implemented macros and aliases for them.
  101. Test cases are created using ``TEST_CASE`` function-like macro. Each test case
  102. must have a brief description written in parentheses, optionally including
  103. custom tags which allow to filter the tests at runtime, such as ``[String]``,
  104. ``[Stress]`` etc.
  105. Test cases are written in a dedicated namespace. This is not required, but
  106. allows to prevent naming collisions for when other static helper functions are
  107. written to accommodate the repeating testing procedures such as populating
  108. common test data for each test, or writing parameterized tests.
  109. Godot supports writing tests per C++ module. For instructions on how to write
  110. module tests, refer to :ref:`doc_custom_module_unit_tests`.
  111. Subcases
  112. ~~~~~~~~
  113. In situations where you have a common setup for several test cases with only slight variations, subcases can be very helpful. Here's an example:
  114. .. code-block:: cpp
  115. TEST_CASE("[SceneTree][Node] Testing node operations with a very simple scene tree") {
  116. // ... common setup (e.g. creating a scene tree with a few nodes)
  117. SUBCASE("Move node to specific index") {
  118. // ... setup and checks for moving a node
  119. }
  120. SUBCASE("Remove node at specific index") {
  121. // ... setup and checks for removing a node
  122. }
  123. }
  124. Each ``SUBCASE`` causes the ``TEST_CASE`` to be executed from the beginning.
  125. Subcases can be nested to an arbitrary depth, but it is advised to limit nesting to no more than one level deep.
  126. Assertions
  127. ~~~~~~~~~~
  128. A list of all commonly used assertions used throughout the Godot tests, sorted
  129. by severity.
  130. +-------------------+----------------------------------------------------------------------------------------------------------------------------------+
  131. | **Assertion** | **Description** |
  132. +-------------------+----------------------------------------------------------------------------------------------------------------------------------+
  133. | ``REQUIRE`` | Test if condition holds true. Fails the entire test immediately if the condition does not hold true. |
  134. +-------------------+----------------------------------------------------------------------------------------------------------------------------------+
  135. | ``REQUIRE_FALSE`` | Test if condition does not hold true. Fails the entire test immediately if the condition holds true. |
  136. +-------------------+----------------------------------------------------------------------------------------------------------------------------------+
  137. | ``CHECK`` | Test if condition holds true. Marks the test run as failing, but allow to run other assertions. |
  138. +-------------------+----------------------------------------------------------------------------------------------------------------------------------+
  139. | ``CHECK_FALSE`` | Test if condition does not hold true. Marks the test run as failing, but allow to run other assertions. |
  140. +-------------------+----------------------------------------------------------------------------------------------------------------------------------+
  141. | ``WARN`` | Test if condition holds true. Does not fail the test under any circumstance, but logs a warning if something does not hold true. |
  142. +-------------------+----------------------------------------------------------------------------------------------------------------------------------+
  143. | ``WARN_FALSE`` | Test if condition does not hold true. Does not fail the test under any circumstance, but logs a warning if something holds true. |
  144. +-------------------+----------------------------------------------------------------------------------------------------------------------------------+
  145. All of the above assertions have corresponding ``*_MESSAGE`` macros, which allow
  146. to print optional message with rationale of what should happen.
  147. Prefer to use ``CHECK`` for self-explanatory assertions and ``CHECK_MESSAGE``
  148. for more complex ones if you think that it deserves a better explanation.
  149. .. seealso::
  150. `doctest: Assertion macros <https://github.com/onqtam/doctest/blob/master/doc/markdown/assertions.md>`_.
  151. Logging
  152. ~~~~~~~
  153. The test output is handled by doctest itself, and does not rely on Godot
  154. printing or logging functionality at all, so it's recommended to use dedicated
  155. macros which allow to log test output in a format written by doctest.
  156. +----------------+-----------------------------------------------------------------------------------------------------------+
  157. | **Macro** | **Description** |
  158. +----------------+-----------------------------------------------------------------------------------------------------------+
  159. | ``MESSAGE`` | Prints a message. |
  160. +----------------+-----------------------------------------------------------------------------------------------------------+
  161. | ``FAIL_CHECK`` | Marks the test as failing, but continue the execution. Can be wrapped in conditionals for complex checks. |
  162. +----------------+-----------------------------------------------------------------------------------------------------------+
  163. | ``FAIL`` | Fails the test immediately. Can be wrapped in conditionals for complex checks. |
  164. +----------------+-----------------------------------------------------------------------------------------------------------+
  165. Different reporters can be chosen at runtime. For instance, here's how the
  166. output can be redirected to an XML file:
  167. .. code-block:: shell
  168. ./bin/<godot_binary> --test --source-file="*test_validate*" --success --reporters=xml --out=doctest.txt
  169. .. seealso::
  170. `doctest: Logging macros <https://github.com/onqtam/doctest/blob/master/doc/markdown/logging.md>`_.
  171. Testing failure paths
  172. ~~~~~~~~~~~~~~~~~~~~~
  173. Sometimes, it's not always feasible to test for an *expected* result. With the
  174. Godot development philosophy of that the engine should not crash and should
  175. gracefully recover whenever a non-fatal error occurs, it's important to check
  176. that those failure paths are indeed safe to execute without crashing the engine.
  177. *Unexpected* behavior can be tested in the same way as anything else. The only
  178. problem this creates is that the error printing shall unnecessarily pollute the
  179. test output with errors coming from the engine itself (even if the end result is
  180. successful).
  181. To alleviate this problem, use ``ERR_PRINT_OFF`` and ``ERR_PRINT_ON`` macros
  182. directly within test cases to temporarily disable the error output coming from
  183. the engine, for instance:
  184. .. code-block:: cpp
  185. TEST_CASE("[Color] Constructor methods") {
  186. ERR_PRINT_OFF;
  187. Color html_invalid = Color::html("invalid");
  188. ERR_PRINT_ON; // Don't forget to re-enable!
  189. CHECK_MESSAGE(html_invalid.is_equal_approx(Color()),
  190. "Invalid HTML notation should result in a Color with the default values.");
  191. }
  192. Special tags in test case names
  193. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  194. These tags can be added to the test case name to modify or extend the test environment:
  195. +-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  196. | **Tag** | **Description** |
  197. +-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  198. | ``[SceneTree]`` | Required for test cases that rely on a scene tree with MessageQueue to be available. It also enables a mock rendering server and :ref:`ThemeDB<class_ThemeDB>`. |
  199. +-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  200. | ``[Editor]`` | Like ``[SceneTree]``, but with additional editor-related infrastructure available, such as :ref:`EditorSettings<class_EditorSettings>`. |
  201. +-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  202. | ``[Audio]`` | Initializes the :ref:`AudioServer<class_AudioServer>` using a mock audio driver. |
  203. +-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  204. | ``[Navigation]`` | Creates the default 2D/3D navigation servers and makes them available for testing. |
  205. +-------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  206. You can use them together to combine multiple test environment extensions.
  207. Testing signals
  208. ~~~~~~~~~~~~~~~
  209. The following macros can be use to test signals:
  210. .. list-table::
  211. :header-rows: 1
  212. :widths: auto
  213. * - Macro
  214. - Description
  215. * - ``SIGNAL_WATCH(object, "signal_name")``
  216. - Starts watching the specified signal on the given object.
  217. * - ``SIGNAL_UNWATCH(object, "signal_name")``
  218. - Stops watching the specified signal on the given object.
  219. * - ``SIGNAL_CHECK("signal_name", Vector<Vector<Variant>>)``
  220. - Checks the arguments of all fired signals. The outer vector contains each fired signal, while the inner vector contains the list of arguments for that signal. The order of signals is significant.
  221. * - ``SIGNAL_CHECK_FALSE("signal_name")``
  222. - Checks if the specified signal was not fired.
  223. * - ``SIGNAL_DISCARD("signal_name")``
  224. - Discards all records of the specified signal.
  225. Below is an example demonstrating the use of these macros:
  226. .. code-block:: cpp
  227. //...
  228. SUBCASE("[Timer] Timer process timeout signal must be emitted") {
  229. SIGNAL_WATCH(test_timer, SNAME("timeout"));
  230. test_timer->start(0.1);
  231. SceneTree::get_singleton()->process(0.2);
  232. Array signal_args;
  233. signal_args.push_back(Array());
  234. SIGNAL_CHECK(SNAME("timeout"), signal_args);
  235. SIGNAL_UNWATCH(test_timer, SNAME("timeout"));
  236. }
  237. //...
  238. Test tools
  239. ----------
  240. Test tools are advanced methods which allow you to run arbitrary procedures to
  241. facilitate the process of manual testing and debugging the engine internals.
  242. These tools can be run by supplying the name of a tool after the ``--test``
  243. command-line option. For instance, the GDScript module implements and registers
  244. several tools to help the debugging of the tokenizer, parser, and compiler:
  245. .. code-block:: shell
  246. ./bin/<godot_binary> --test gdscript-tokenizer test.gd
  247. ./bin/<godot_binary> --test gdscript-parser test.gd
  248. ./bin/<godot_binary> --test gdscript-compiler test.gd
  249. If any such tool is detected, then the rest of the unit tests are skipped.
  250. Test tools can be registered anywhere throughout the engine as the registering
  251. mechanism closely resembles of what doctest provides while registering test
  252. cases using dynamic initialization technique, but usually these can be
  253. registered at corresponding ``register_types.cpp`` sources (per module or core).
  254. Here's an example of how GDScript registers test tools in
  255. ``modules/gdscript/register_types.cpp``:
  256. .. code-block:: cpp
  257. #ifdef TESTS_ENABLED
  258. void test_tokenizer() {
  259. TestGDScript::test(TestGDScript::TestType::TEST_TOKENIZER);
  260. }
  261. void test_parser() {
  262. TestGDScript::test(TestGDScript::TestType::TEST_PARSER);
  263. }
  264. void test_compiler() {
  265. TestGDScript::test(TestGDScript::TestType::TEST_COMPILER);
  266. }
  267. REGISTER_TEST_COMMAND("gdscript-tokenizer", &test_tokenizer);
  268. REGISTER_TEST_COMMAND("gdscript-parser", &test_parser);
  269. REGISTER_TEST_COMMAND("gdscript-compiler", &test_compiler);
  270. #endif
  271. The custom command-line parsing can be performed by a test tool itself with the
  272. help of OS :ref:`get_cmdline_args<class_OS_method_get_cmdline_args>` method.
  273. Integration tests for GDScript
  274. ------------------------------
  275. Godot uses doctest to prevent regressions in GDScript during development. There
  276. are several types of test scripts which can be written:
  277. - tests for expected errors;
  278. - tests for warnings;
  279. - tests for features.
  280. Therefore, the process of writing integration tests for GDScript is the following:
  281. 1. Pick a type of a test script you'd like to write, and create a new GDScript
  282. file under the ``modules/gdscript/tests/scripts`` directory within
  283. corresponding sub-directory.
  284. 2. Write GDScript code. The test script must have a function called ``test()``
  285. which takes no arguments. Such function will be called by the test runner.
  286. The test should not have any dependency unless it's part of the test too.
  287. Global classes (using ``class_name``) are registered before the runner
  288. starts, so those should work if needed.
  289. Here's an example test script:
  290. ::
  291. func test():
  292. if true # Missing colon here.
  293. print("true")
  294. 3. Change directory to the Godot source repository root.
  295. .. code-block:: shell
  296. cd godot
  297. 4. Generate ``*.out`` files to update the expected results from the output:
  298. .. code-block:: shell
  299. bin/<godot_binary> --gdscript-generate-tests modules/gdscript/tests/scripts
  300. You may add the ``--print-filenames`` option to see filenames as their test
  301. outputs are generated. If you are working on a new feature that is causing
  302. hard crashes, you can use this option to quickly find which test file causes
  303. the crash and debug from there.
  304. 5. Run GDScript tests with:
  305. .. code-block:: shell
  306. ./bin/<godot_binary> --test --test-suite="*GDScript*"
  307. This also accepts the ``--print-filenames`` option (see above).
  308. If no errors are printed and everything goes well, you're done!
  309. .. warning::
  310. Make sure the output does have the expected values before submitting a pull
  311. request. If ``--gdscript-generate-tests`` produces ``*.out`` files which are
  312. unrelated to newly added tests, you should revert those files back and
  313. only commit ``*.out`` files for new tests.
  314. .. note::
  315. The GDScript test runner is meant for testing the GDScript implementation,
  316. not for testing user scripts nor testing the engine using scripts. We
  317. recommend writing new tests for already resolved
  318. `issues related to GDScript at GitHub <https://github.com/godotengine/godot/issues?q=is%3Aissue+label%3Atopic%3Agdscript+is%3Aclosed>`_,
  319. or writing tests for currently working features.
  320. .. note::
  321. If your test case requires that there is no ``test()``
  322. function present inside the script file,
  323. you can disable the runtime section of the test by naming the script file so that it matches the pattern ``*.notest.gd``.
  324. For example, "test_empty_file.notest.gd".