mte/unikernel/duniverse/dune_/doc/tests.rst

.. _writing-tests:

*************************
Writing and Running Tests
*************************

.. TODO(diataxis)

   This is mostly a guide, or rather several of them. There is also some
   reference in it.

   Something we can do is split this into:

   - one how-to guide per test technique
   - a "choosing a test technique" how-to guide
   - reference for ``inline_tests.backend``
   - reference for cram tests

Dune tries to streamline the testing story as much as possible, so
you can focus on the tests themselves and not bother with setting
up various test frameworks.

In this section, we'll explain the workflow to deal with tests in Dune. In
particular, we'll see how to run the test suite of a project, how to describe
your tests to Dune, and how to promote test results as expectation.

We distinguish three kinds of tests:

* Inline tests - written directly inside the ``.ml`` files of a library

* Custom tests - run an executable, possibly followed by an action such as
  diffing the produced output.

* Cram tests - expect tests written in Cram_ style.


Running Tests
=============

Whatever the tests of a project are, the usual way to run tests with Dune is to
call ``dune runtest`` from the shell (or the command alias ``dune test``). This
will run all the tests defined in the current directory and any subdirectory
recursively.

Note that in any case, ``dune runtest`` is simply shorthand for building the
``runtest`` alias, so you can always ask Dune to run the tests in conjunction
with other targets by passing ``@runtest`` to ``dune build``. For instance:

.. code:: console

   $ dune build @install @runtest
   $ dune build @install @test/runtest


Running a Single Test
---------------------

If you would only like to run a single test for your project, you may use ``dune
exec`` to run the test executable (for the sake of this example,
``project/tests/myTest.ml``):

.. code:: console

  $ dune exec project/tests/myTest.exe

To run :ref:`cram-tests` you can pass their paths to the ``dune test``  command.

.. code:: console

   $ dune test tests/myCramTest.t

This works both for directory and file cram tests.

Running Tests in a Directory
----------------------------

You can also pass a directory argument to run the tests from a subtree. For
instance, ``dune runtest test`` will only run the tests from the ``test``
directory and any subdirectory of ``test`` recursively.

.. _inline_tests:


Inline Tests
============

There are several inline tests frameworks available for OCaml, such as
ppx_inline_test_ and qtest_. We will use ppx_inline_test_ as an
example because it has the necessary setup to be used with Dune out of the box.

ppx_inline_test_ allows one to write tests directly inside ``.ml`` files as
follows:


.. code:: ocaml


   let rec fact n = if n = 1 then 1 else n * fact (n - 1)

   let%test _ = fact 5 = 120

The file must be preprocessed with the ``ppx_inline_test`` PPX rewriter,
so for instance the ``dune`` file might look like this:

.. code:: dune

   (library
    (name foo)
    (preprocess (pps ppx_inline_test)))

In order to tell Dune that our library contains inline tests, 
we have to add an ``inline_tests`` field:

.. code:: dune

   (library
    (name foo)
    (inline_tests)
    (preprocess (pps ppx_inline_test)))

We can now build and execute this test by running ``dune runtest``. For
instance, if we make the test fail by replacing ``120`` by ``0`` we get:

.. code:: console

   $ dune runtest
   [...]
   File "src/fact.ml", line 3, characters 0-25: <<(fact 5) = 0>> is false.

   FAILED 1 / 1 tests

Every inline test library generates an alias with the library name prefixed by
`runtest-`. You can build the specific inline test library by running
``dune build @runtest-foo`` in this case.

Note that in this case Dune knew how to build and run the tests
without any special configuration. This is because ``ppx_inline_test``
defines an inline tests backend that's used by the library. Some
other frameworks, such as qtest_, don't have any special library or PPX
rewriter. To use such a framework, you must tell Dune about it,
as it cannot guess. You can do that by adding a ``backend``
field:

.. code:: dune

   (library
    (name foo)
    (inline_tests (backend qtest.lib)))

In the example above, the name `qtest.lib` comes from the `public_name` field
in `qtest`'s own `dune` file.

Note that using ``ppx_inline_test`` requires that the opam package
``ppx_inline_test`` be installed in your switch. If you use ``ppx_inline_test``
in a package then that package must `unconditionally` depend on
``ppx_inline_test`` (ie. ``ppx_inline_test`` can't be a ``with-test``
dependency).

Inline Expectation Tests
------------------------

Inline expectation tests are a special case of inline tests where written OCaml code 
prints something followed by what you expect this code to print. 
For instance, using ppx_expect_:

.. code:: ocaml

   let%expect_test _ =
     print_endline "Hello, world!";
     [%expect{|
       Hello, world!
     |}]

The test procedure consist of executing the OCaml code and replacing
the contents of the ``[%expect]`` extension point by the real
output. You then get a new file that you can compare to the original
source file. Expectation tests are a neat way to write tests as the
following test elements are clearly identified:

- The code of the test
- The test expectation
- The test outcome

You can have a look at `this blog post
<https://blog.janestreet.com/testing-with-expectations/>`_ to find out
more about expectation tests. To Dune, the workflow for
expectation tests is always as follows:

- Write the test with some empty expect nodes in it
- Run the tests
- Check the suggested correction and promote it as the original source
  file if you are happy with it

Dune makes this workflow very easy. Simply add ``ppx_expect`` to
your list of PPX rewriters as follows:

.. code:: dune

   (library
    (name foo)
    (inline_tests)
    (preprocess (pps ppx_expect)))

Then calling ``dune runtest`` will run these tests, and in case of
mismatch, Dune will print a diff of the original source file and
the suggested correction. For instance:

.. code:: console

   $ dune runtest
   [...]
   -src/fact.ml
   +src/fact.ml.corrected
   File "src/fact.ml", line 5, characters 0-1:
   let rec fact n = if n = 1 then 1 else n * fact (n - 1)

   let%expect_test _ =
     print_int (fact 5);
   -  [%expect]
   +  [%expect{| 120 |}]

In order to accept the correction, simply run:

.. code:: console

   $ dune promote

You can also make Dune automatically accept the correction after
running the tests by typing:

.. code:: console

   $ dune runtest --auto-promote

Finally, some editor integration can make the editor do the
promotion, which in turn makes the workflow even smoother.


Running a Subset of the Test Suite
----------------------------------

You may also run a group of tests located under a directory with:

.. code:: console

  $ dune runtest mylib/tests

The above command will run all tests defined in tests and its subdirectories.


Running Tests in Bytecode or JavaScript
---------------------------------------

By default, Dune runs inline tests in native mode, unless native
compilation isn't available. In which case, it runs them in bytecode.
You can change this setting to choose the modes that tests should run
in. To do this, add a ``modes`` field to the ``inline_tests``
field. Available modes are:

- ``byte`` for running tests in byte code
- ``native`` for running tests in native mode
- ``best`` for running tests in native mode with fallback to byte code,
  if native compilation is not available
- ``js`` for running tests in JavaScript using Node.js
- ``wasm`` for running tests in Wasm using Node.js

For instance:

.. code:: ocaml

   (library
    (name foo)
    (inline_tests (modes byte best js wasm))
    (preprocess (pps ppx_expect)))


Specifying Inline Test Dependencies
-----------------------------------

If your tests are reading files, you must tell Dune by adding
a ``deps`` field the ``inline_tests`` field. The argument of this
``deps`` field follows the usual :doc:`concepts/dependency-spec`. For instance:

.. code:: ocaml

   (library
    (name foo)
    (inline_tests (deps data.txt))
    (preprocess (pps ppx_expect)))


Passing Special Arguments to the Test Runner
--------------------------------------------

Under the hood, a test executable is built by Dune. Depending on
the backend used, this runner might take useful command line
arguments. You can specify such flags by using a ``flags`` field, such
as:

.. code:: ocaml

   (library
    (name foo)
    (inline_tests (flags (-foo bar)))
    (preprocess (pps ppx_expect)))

The argument of the ``flags`` field follows the
:doc:`reference/ordered-set-language`.


Passing Special Arguments to the Test Executable
------------------------------------------------

To control how the test executable is built, it's possible to customize a subset
of compilation options for an executable using the ``executable`` field. Dune
gives you this ability by simply specifying command line arguments as flags.
You can specify such flags by using ``flags`` field. For instance:

.. code:: ocaml

   (library
    (name foo)
    (inline_tests
     (flags (-foo bar)
     (executable
      (flags (-foo bar))))
     (preprocess (pps ppx_expect))))

The argument of the ``flags`` field follows the :doc:`reference/ordered-set-language`.

Using Additional Libraries in the Test Runner
---------------------------------------------

When tests are not part of the library code, it's possible that tests
require additional libraries than the library being tested. This is
the case with qtest_, as tests are written in comments. You can specify
such libraries using a ``libraries`` field, such as:

.. code:: ocaml

   (library
    (name foo)
    (inline_tests
     (backend qtest)
     (libraries bar)))


Changing the Flags of the Linking Step of the Test Runner
---------------------------------------------------------

You can use the ``link_flags`` field to change the linker flags
passed to ``ocamlopt`` when building the test runner. By default, the
linking flags are ``-linkall``. You probably want to keep
``-linkall`` as one of the new list of flags (unless you know what you
are doing), forcing the linker to load your test module, since the test
runner doesn't depend on anything itself. This field supports
``(:include ...)`` forms.

.. code:: dune

   (library
    (name foo)
    (inline_tests
     (executable
      (link_flags -linkall -noautolink -cclib -Wl,-Bstatic -cclib -lm)))
    (preprocess (pps ppx_expect)))


Defining Your Own Inline Test Backend
-------------------------------------

If you are writing a test framework (or for other specific cases), you might
want to define your own inline tests backend. If your framework is
naturally implemented by a library or PPX rewriter that's necessary to write tests, 
you should define this library as a backend. Otherwise simply create an 
empty library with your chosen backend's name.

In order to define a library as an inline tests backend, simply add an
``inline_tests.backend`` field to the library stanza. An inline tests
backend is specified by four parameters:

1. How to create the test runner
2. How to build the test runner
3. How to run the test runner
4. Optionally how to run the test runner to list partitions

These four parameters can be specified inside the
``inline_tests.backend`` field, which accepts the following fields:

.. code:: dune

   (generate_runner       <action>)
   (runner_libraries     (<ocaml-libraries>))
   (flags                 <flags>)
   (list_partitions_flags <flags>)
   (extends              (<backends>))

For instance:

``<action>`` follows the :doc:`reference/actions/index` specification. It
describes an action that should be executed in the library's directory using
this backend for their tests. It's expected that the action will produce some
OCaml code on its standard output. This code will constitute the test runner.
The action can use the following additional variables:

- ``%{library-name}`` --- the name of the library being tested
- ``%{impl-files}`` --- the list of implementation files in the
  library, i.e., all the ``.ml`` and ``.re`` files
- ``%{intf-files}`` --- the list of interface files in the library,
  i.e., all the ``.mli`` and ``.rei`` files

The ``runner_libraries`` field specifies what OCaml libraries the test
runner uses. For instance, if the ``generate_runner`` actions
generates something like ``My_test_framework.runtests ()``, then you
should probably put ``my_test_framework`` in the ``runner_libraries``
field.

If your test runner needs specific flags, you should pass them in the
``flags`` field. You can use the ``%{library-name}`` variable in this
field.

If your test runner supports test partitions, you should pass the
flags necessary for listing partitions in the
``list_partitions_flags`` field. In such scenario, the ``flags`` field
will also accepts a ``%{partition}`` variable.

Finally, a backend can be an extension of another backend. In this
case, you must specify this in the ``extends`` field. For instance,
ppx_expect_ is an extension of ppx_inline_test_. It's possible to use
a backend with several extensions in a library; however, there must be
exactly one *root backend*, i.e., exactly one backend that isn't an
extension of another one.

When using a backend with extensions, the various fields are simply
concatenated. The order in which they are concatenated is unspecified;
however, if a backend ``b`` extends a backend ``a``, then ``a`` will
always come before ``b``.


Example of Backend
~~~~~~~~~~~~~~~~~~

In this example, we put tests in comments of the form:

.. code:: ocaml

   (*TEST: assert (fact 5 = 120) *)

The backend for such a framework looks like this:

.. code:: dune

   (library
    (name simple_tests)
    (inline_tests.backend
     (generate_runner (run sed "s/(\\*TEST:\\(.*\\)\\*)/let () = \\1;;/" %{impl-files}))))

Now all you have to do is write ``(inline_tests ((backend
simple_tests)))`` wherever you want to write such tests. Note that
this is only an example. We don't recommend using ``sed`` in your
build, as this would cause portability problems.


Custom Tests
============

We said in `Running tests`_ that to run tests, Dune simply builds
the ``runtest`` alias. As a result, you simply need to add an action 
to this alias in any directory in order to define custom tests. For instance, if
you have a binary ``tests.exe`` that you want to run as part of
running your test suite, simply add this to a ``dune`` file:

.. code:: dune

   (rule
    (alias  runtest)
    (action (run ./tests.exe)))

Hence to define a test, a pair of alias and executable stanzas are required.
To simplify this common pattern, Dune provides a :ref:`tests-stanza` stanza to
define multiple tests and their aliases at once:

.. code:: dune

   (tests (names test1 test2))


Diffing the Result
------------------

It's often the case that we want to compare the actual output of a test to
an expected one. For that, Dune offers the ``diff`` command,
which in essence is the same as running the ``diff`` tool, except that
it's more integrated in Dune, especially with the ``promote``
command. For instance, let's consider this test:

.. code:: dune

   (rule
   (with-stdout-to tests.output (run ./tests.exe)))

   (rule
    (alias runtest)
    (action (diff tests.expected tests.output)))

After having run ``tests.exe`` and dumping its output to ``tests.output``, Dune
will compare the latter to ``tests.expected``. In case of mismatch, Dune will
print a diff and then the ``dune promote`` command can be used to copy over the
generated ``test.output`` file to ``tests.expected`` in the source tree.

Alternatively, the :ref:`tests-stanza` also supports this style of tests.

.. code:: dune

   (tests (names tests))

Dune expects the existence of a ``tests.expected`` file to infer that this is an
expected test.

This provides a nice way of dealing with the usual *write code*,
*run*, and *promote* cycle of testing. For instance:

.. code:: console

    $ dune runtest
    [...]
    -tests.expected
    +tests.output
    File "tests.expected", line 1, characters 0-1:
    -Hello, world!
    +Good bye!
    $ dune promote
    Promoting _build/default/tests.output to tests.expected.

Note that if available, the diffing is done using the patdiff_ tool,
which displays nicer looking diffs than the standard ``diff``
tool. You can change that by passing ``--diff-command CMD`` to
Dune.

.. _cram-tests:


Cram Tests
==========

Cram tests are expectation tests written in a shell-like syntax. They are ideal
for testing binaries. Cram tests are automatically discovered from files or directories
with a ``.t`` extension. By default, this has been enabled since Dune 3.0. For
older versions, it must be manually enabled in the ``dune-project`` file:

.. code:: dune

   (lang dune 2.7)
   (cram enable)


File Tests
----------

To define a standalone test, we create a ``.t`` file. For example, ``foo.t``:

.. code:: cram

   Simplest possible Cram test
     $ echo "testing"

This simple example demonstrates two components of Cram tests: comments and
commands. See :doc:`reference/cram` for a description of the syntax.

To run the test and promote the results:

.. code:: console

   $ dune runtest
   $ dune promote

We now see the output of the command:

.. code:: cram

   Simplest possible cram test
     $ echo "testing"
     testing

This is the main advantage of expect tests. We don't need to write assertions
manually; instead we detect failure when the command produces a different output
than what is recorded in the test script.

For example, here's an example of how we'd test the ``wc`` utility. ``wc.t``:

.. code:: cram

   We create a test artifact called "foo"
     $ cat >foo <<EOF
     > foo
     > bar
     > baz
     > EOF

   After creating the fixture, we want to verify that ``wc`` gives us the right
   result:
     $ wc -l foo | awk '{ print $1 }'
     4

The above example uses the doc syntax, piping the subsequent lines to
``cat``. This is convenient for creating small test artifacts.


Directory Tests
---------------

In the above example we used ``cat`` to create the test artifact, but what if
there are too many artifacts to comfortably fit in test file? Or some of the
artifacts are binary? 

It's possible to include the artifacts as normal files or
directories, provided the test is defined as a directory. The name of the test
directory must end with ``.t`` and must include a ``run.t`` as the test script.
Everything else in that directory is treated as raw data for the test. It's not
possible to define rules using ``dune`` files in such a directory.

We convert the ``wc`` test above into a directory test ``wc.t``:

.. code:: console

   $ ls wc.t
   run.t foo.txt bar/

This defines a directory test ``wc.t`` which must include a ``run.t`` file as
the test script, with ``fool.txt`` and ``bar`` are test artifacts. We may then
access their contents in the test script ``run.t``:

.. code:: cram

   Testing wc:
     $ wc -l foo | awk '{ print $1 }'
     4
     $ wc -l $(ls bar) | awk '{ print $1 }'
     1231

.. seealso:: :doc:`(cram) stanza reference </reference/dune/cram>`

Testing an OCaml Program
------------------------

The most common testing situation involves testing an executable that is defined
in Dune. For example:

.. code:: dune

   (executable
    (name wc)
    (public_name wc))

To use this binary in the Cram test, we should depend on the binary in the test:

.. code::

   (cram
    (deps %{bin:wc}))


Sandboxing
----------

Since Cram tests often create intermediate artifacts, it's important that Cram
tests are executed in a clean environment. This is why all Cram tests are
sandboxed. To respect sandboxing, every test should specify dependency on any
artifact that might rely on using the ``deps`` field.

See :doc:`concepts/sandboxing` for details about the sandboxing mechanism.


Test Output Sanitation
----------------------

In some situations, Cram tests emit non portable or non-deterministic output. We
recommend sanitising such outputs using pipes. For example, we can scrub the
OCaml magic number using ``sed`` as follows:

.. code:: console

   $ ocamlc -config | grep "cmi_magic_number:" | sed 's/Caml.*/$SPECIAL_CODE/'
   cmi_magic_number: $SPECIAL_CODE

By default, Dune will scrub some paths from the output of the tests. The
default list of paths is:

- The ``PWD`` of the test will be replaced by ``$TESTCASE_ROOT``
- The temporary directory for the current script will be replaced by ``$TMPDIR``

To add additional paths to this sanitation mechanism, it's sufficient to modify
the standard BUILD_PATH_PREFIX_MAP_ environment variable. For example:

.. code:: console

   $ export BUILD_PATH_PREFIX_MAP="HOME=$HOME:$BUILD_PATH_PREFIX_MAP"
   $ echo $HOME
   $HOME

Note: Unlike Dune's version of Cram, the original specification for Cram
supports regular expression and glob filtering for matching output. We chose
not to implement this feature because it breaks the test, diff, and accept cycle.
With regex or glob matching, the output must now be manually inspected and
possibly updated. We consider the postprocessing approach described here as
superior and will not introduce output matchers.

.. _ppx_inline_test:       https://github.com/janestreet/ppx_inline_test
.. _ppx_expect:            https://github.com/janestreet/ppx_expect
.. _qtest:                 https://github.com/vincent-hugot/qtest
.. _patdiff:               https://github.com/janestreet/patdiff
.. _cram:                  https://bitheap.org/cram/
.. _BUILD_PATH_PREFIX_MAP: https://reproducible-builds.org/specs/build-path-prefix-map/