8.3 KiB
Rule production
This document describes how rule production works in Dune. It was originally written by Jérémie Dimino as part of the streaming RFC, but moved into the dev documentation as it provides a great overview on how this part of Dune works at present.
How does rule production works?
Dune_engine.Load_rules
The production of rules is driven by the module Load_rules in the
dune_engine library. This library is the build system core of
Dune. It is meant as a general purpose library for writing build
systems, and the Dune software is built on top of it. In theory,
dune_engine shouldn't know about dune or dune-project
files. However, for historical reason this is not the case yet and
dune_engine still knows some things about them.
As we work on Dune, we expect that dune_engine will become more and
more agnostic. Even though it is not completely agnostic, we have
successfully been using it to build Jane Street code base, using the
Jane Street rules on top of this core. So it's already more general
than Dune itself.
For the purpose of this design doc, we will treat dune_engine as a
completely general library that doesn't know about dune files.
The main feature of Load_rules is the Load_rules.load_dir function:
val load_dir : dir:Path.t -> Loaded.t Memo.t
Loaded.t represents a "loaded" set of rules for a particular
directory. It can also be thought as a "compiled" set of rules. A
Loaded.t contains all the rules in existence that produce targets in
dir. For instance, given a Loaded.t we can figure out all the
files that would be produced in dir if we were building everything
that could be built. While dir is a build directory, this also
includes files present in the source tree. This is because
Load_rules.load_dir implicitly adds copy rules for all source files
present in the source directory that correspond to the build directory
dir. For instance, if dir is _build/default/src, Dune will
implicitly add rules to copy files in src to _build/default/src.
Except for rules that have the special promote or fallback modes.
This is in fact how Dune evaluates globs during the build. Indeed,
when writing dune files we work in an imaginary world where both the
source files and the generated files are present. So when we write
(deps (glob_files *.txt)), this *.txt denotes both .txt files
that are present on disk in the source tree but also as the ones that
can be generated by the build.
In practice, to evaluate (glob_files *.txt) in directory d, Dune
calls Load_rules.load_dir ~dir:d and filter the list of files that can
be built. Similarly, when Dune needs to build a file
_build/default/src/x, it first calls Load_rules.load_dir with
_build/default/src and then looks up a rule that has x has
target in the returned Loaded.t. The Load_rules.load_dir is
memoised, so it can be called multiple times during the build without
guilt.
While Load_rules is responsible for driving the production of rules,
it is part of dune_engine which doesn't know about dune files and
doesn't know about OCaml libraries or OCaml compilation in general. So
it is not responsible for actually producing the build rules that
allow to build Dune projects. Instead, Load_rules defers the actual
production of rules to a callback that it obtains via
Build_config. Inside Dune, this callback is implemented by the
Gen_rules module inside the dune_rules library. dune_rules is
the library that is responsible for parsing, interpreting and
compiling dune files down to low-level build rules.
Dune_rules.Gen_rules
The entry of Dune_rules.Gen_rules is the gen_rules function. Its
API looks like:
val gen_rules :
Build_config.Context_or_install.t ->
dir:Path.Build.t ->
string list ->
Build_config.gen_rules_result Memo.t
Where Build_config.gen_rules_result is, in most cases —when the value
returned is Build_config.Rules _—, a "raw" set of rules. Raw in the sense
that there is no overlap checks or any other checks. During the rule
production phase, we merely accumulate a set of rules that is later
processed. The API of gen_rules is in fact a bit more complex, but
the above definition is enough for the purpose of this document.
The first thing gen_rules does is analyse the directory it is
given. If the directory corresponds to a source directory with a dune
file, gen_rules will dispatch the call to the part of dune_rules
that parses and interprets the dune file. This is the simplest case,
but even in this case there are some things worth mentioning.
For instance, when compiling an OCaml library dune stores the
artifacts for the library in generated dot-directories. For instance,
the cmi files for library foo living in source directory src will
end up in _build/default/src/.foo.objs/byte. We could produce these
rules when gen_rules is called with directory
_build/default/src/.foo.objs/byte, however that would spread out the
logic for interpreting library stanzas. It is much simpler to
produce all the build rules corresponding to a library stanza in one
go. This is what is happening at the moment: when called with
directory _build/default/src, gen_rules will not only produce
rules for this directory but will also produce rules for
_build/default/src/.foo.objs/byte and various other directories.
Load_rules doesn't know anything about this. And in particular, it
doesn't know that it is the gen_rules call for directory
_build/default/src/ that will produce the rules for the dot
subdirectories. When Load_rules loads the rules for the
.../.foo.objs/byte sub-directory, it simply calls gen_rules with
this directory. It is gen_rules that "redirects" the call to the
_build/default/src directory by calling
Load_rules.load_dir_and_produce_its_rules. This function simply
calls Load_rules.load_dir and re-emits all the raw rules that were
returned by the corresponding gen_rules call.
This works because Load_rules.load_dir accepts the facts that
gen_rules produces rules for many directory at once. It simply
filters out the result. But for things to behave well, the unwritten
following invariant must hold: gen_rules ~dir:d is allowed to
generate rules for directory d' iff gen_rules ~dir:d' emits a call
to Load_rules.load_dir ~dir:d.
This scenario happens in a number of cases. All these cases share a common pattern: the redirections are always to an ancestor directory. At the moment, there is one exception to this pattern in the odoc rules, however it is easy to remove.
Finally, the copy_files stanza creates another form of dependency
between directory. In order to calculate the targets produced by
copy_files, which needs to be known at rule production time, we need
to evaluate the glob given to copy_files. Which requires doing a
call to Load_rules.load_dir as previously described. Contrary to the
other form of dependency we just describe, this ones can go from any
directory to any other directory. For instance, the following stanza
in src/dune:
(copy_files foo/*.txt)
would create a dependency from _build/default/src to
_build_default/src/foo.
So in the end, if we were looking at the internal computation graph of
Dune and narrowing it to just the calls to Load_rules.load_dir, we
would see a graph with many edges going from a directory to one of its
ancestor. These would mostly be between generated dot-subdirectories
and their first ancestor that has a corresponding directory in the
source tree. Plus a few other arbitrary ones for each copy_files
stanza.
Directory targets
Before directory targets, answering the question "what rules produces
file X?" was easy. Dune would just call Load_rules.load_dir and
lookup X in the result. With directory targets, things are a bit
more complicated. Indeed, X might also be produced by a directory
target in an ancestor directory. This means that Load_rules.load_dir
now need to look in parent directories as well, which introduce more
dependencies from directories to their parents and can create cycles
because of copy_files stanza that create dependencies in the other
direction.
At a result, some combinations of copy_files and directory targets
don't produce the expected result. This is documented in the test
suite.