mte/unikernel/duniverse/dune_/doc/dev/rule-production.md

# Rule production

This document describes how rule production works in Dune. It was originally
written by Jérémie Dimino as part of the
[streaming RFC](https://github.com/ocaml/dune/pull/5251), but moved
into the dev documentation as it provides a great overview on how this part of
Dune works at present.

## How does rule production works?

### `Dune_engine.Load_rules`

The production of rules is driven by the module `Load_rules` in the
`dune_engine` library. This library is the build system core of
Dune. It is meant as a general purpose library for writing build
systems, and the Dune software is built on top of it. In theory,
`dune_engine` shouldn't know about `dune` or `dune-project`
files. However, for historical reason this is not the case yet and
`dune_engine` still knows some things about them.

As we work on Dune, we expect that `dune_engine` will become more and
more agnostic. Even though it is not completely agnostic, we have
successfully been using it to build Jane Street code base, using the
Jane Street rules on top of this core. So it's already more general
than Dune itself.

For the purpose of this design doc, we will treat `dune_engine` as a
completely general library that doesn't know about `dune` files.

The main feature of `Load_rules` is the `Load_rules.load_dir` function:

```ocaml
val load_dir : dir:Path.t -> Loaded.t Memo.t
```

`Loaded.t` represents a "loaded" set of rules for a particular
directory. It can also be thought as a "compiled" set of rules. A
`Loaded.t` contains all the rules in existence that produce targets in
`dir`. For instance, given a `Loaded.t` we can figure out all the
files that would be produced in `dir` if we were building everything
that could be built. While `dir` is a build directory, this also
includes files present in the source tree. This is because
`Load_rules.load_dir` implicitly adds copy rules for all source files
present in the source directory that correspond to the build directory
`dir`. For instance, if `dir` is `_build/default/src`, Dune will
implicitly add rules to copy files in `src` to `_build/default/src`.
Except for rules that have the special `promote` or `fallback` modes.

This is in fact how Dune evaluates globs during the build. Indeed,
when writing `dune` files we work in an imaginary world where both the
source files and the generated files are present. So when we write
`(deps (glob_files *.txt))`, this `*.txt` denotes both `.txt` files
that are present on disk in the source tree but also as the ones that
can be generated by the build.

In practice, to evaluate `(glob_files *.txt)` in directory `d`, Dune
calls `Load_rules.load_dir ~dir:d` and filter the list of files that can
be built. Similarly, when Dune needs to build a file
`_build/default/src/x`, it first calls `Load_rules.load_dir` with
`_build/default/src` and then looks up a rule that has `x` has
target in the returned `Loaded.t`. The `Load_rules.load_dir` is
memoised, so it can be called multiple times during the build without
guilt.

While `Load_rules` is responsible for driving the production of rules,
it is part of `dune_engine` which doesn't know about `dune` files and
doesn't know about OCaml libraries or OCaml compilation in general. So
it is not responsible for actually producing the build rules that
allow to build Dune projects. Instead, `Load_rules` defers the actual
production of rules to a callback that it obtains via
`Build_config`. Inside Dune, this callback is implemented by the
`Gen_rules` module inside the `dune_rules` library. `dune_rules` is
the library that is responsible for parsing, interpreting and
compiling `dune` files down to low-level build rules.

### `Dune_rules.Gen_rules`

The entry of `Dune_rules.Gen_rules` is the `gen_rules` function. Its
API looks like:

```ocaml
val gen_rules :
  Build_config.Context_or_install.t ->
  dir:Path.Build.t ->
  string list ->
  Build_config.gen_rules_result Memo.t
```

Where `Build_config.gen_rules_result` is, in most cases —when the value
returned is `Build_config.Rules _`—, a "raw" set of rules. Raw in the sense
that there is no overlap checks or any other checks. During the rule
production phase, we merely accumulate a set of rules that is later
processed. The API of `gen_rules` is in fact a bit more complex, but
the above definition is enough for the purpose of this document.

The first thing `gen_rules` does is analyse the directory it is
given. If the directory corresponds to a source directory with a `dune`
file, `gen_rules` will dispatch the call to the part of `dune_rules`
that parses and interprets the `dune` file. This is the simplest case,
but even in this case there are some things worth mentioning.

For instance, when compiling an OCaml library dune stores the
artifacts for the library in generated dot-directories. For instance,
the cmi files for library `foo` living in source directory `src` will
end up in `_build/default/src/.foo.objs/byte`. We could produce these
rules when `gen_rules` is called with directory
`_build/default/src/.foo.objs/byte`, however that would spread out the
logic for interpreting `library` stanzas. It is much simpler to
produce all the build rules corresponding to a `library` stanza in one
go. This is what is happening at the moment: when called with
directory `_build/default/src`, `gen_rules` will not only produce
rules for this directory but will also produce rules for
`_build/default/src/.foo.objs/byte` and various other directories.

`Load_rules` doesn't know anything about this. And in particular, it
doesn't know that it is the `gen_rules` call for directory
`_build/default/src/` that will produce the rules for the dot
subdirectories. When `Load_rules` loads the rules for the
`.../.foo.objs/byte` sub-directory, it simply calls `gen_rules` with
this directory. It is `gen_rules` that "redirects" the call to the
`_build/default/src` directory by calling
`Load_rules.load_dir_and_produce_its_rules`. This function simply
calls `Load_rules.load_dir` and re-emits all the raw rules that were
returned by the corresponding `gen_rules` call.

This works because `Load_rules.load_dir` accepts the facts that
`gen_rules` produces rules for many directory at once. It simply
filters out the result. But for things to behave well, the unwritten
following invariant must hold: `gen_rules ~dir:d` is allowed to
generate rules for directory `d'` iff `gen_rules ~dir:d'` emits a call
to `Load_rules.load_dir ~dir:d`.

This scenario happens in a number of cases. All these cases share a
common pattern: the redirections are always to an ancestor
directory. At the moment, there is one exception to this pattern in
the odoc rules, however it is easy to remove.

Finally, the `copy_files` stanza creates another form of dependency
between directory. In order to calculate the targets produced by
`copy_files`, which needs to be known at rule production time, we need
to evaluate the glob given to `copy_files`. Which requires doing a
call to `Load_rules.load_dir` as previously described. Contrary to the
other form of dependency we just describe, this ones can go from any
directory to any other directory. For instance, the following stanza
in `src/dune`:

```
(copy_files foo/*.txt)
```

would create a dependency from `_build/default/src` to
 `_build_default/src/foo`.

So in the end, if we were looking at the internal computation graph of
Dune and narrowing it to just the calls to `Load_rules.load_dir`, we
would see a graph with many edges going from a directory to one of its
ancestor. These would mostly be between generated dot-subdirectories
and their first ancestor that has a corresponding directory in the
source tree. Plus a few other arbitrary ones for each `copy_files`
stanza.

## Directory targets

Before directory targets, answering the question "what rules produces
file X?" was easy. Dune would just call `Load_rules.load_dir` and
lookup `X` in the result. With directory targets, things are a bit
more complicated. Indeed, `X` might also be produced by a directory
target in an ancestor directory. This means that `Load_rules.load_dir`
now need to look in parent directories as well, which introduce more
dependencies from directories to their parents and can create cycles
because of `copy_files` stanza that create dependencies in the other
direction.

At a result, some combinations of `copy_files` and directory targets
don't produce the expected result. This is documented in the test
suite.