unikernel/duniverse/dune_/doc/dev/rule-production.md

@@ -0,0 +1,176 @@
+# Rule production
+
+This document describes how rule production works in Dune. It was originally
+written by Jérémie Dimino as part of the
+[streaming RFC](https://github.com/ocaml/dune/pull/5251), but moved
+into the dev documentation as it provides a great overview on how this part of
+Dune works at present.
+
+## How does rule production works?
+
+### `Dune_engine.Load_rules`
+
+The production of rules is driven by the module `Load_rules` in the
+`dune_engine` library. This library is the build system core of
+Dune. It is meant as a general purpose library for writing build
+systems, and the Dune software is built on top of it. In theory,
+`dune_engine` shouldn't know about `dune` or `dune-project`
+files. However, for historical reason this is not the case yet and
+`dune_engine` still knows some things about them.
+
+As we work on Dune, we expect that `dune_engine` will become more and
+more agnostic. Even though it is not completely agnostic, we have
+successfully been using it to build Jane Street code base, using the
+Jane Street rules on top of this core. So it's already more general
+than Dune itself.
+
+For the purpose of this design doc, we will treat `dune_engine` as a
+completely general library that doesn't know about `dune` files.
+
+The main feature of `Load_rules` is the `Load_rules.load_dir` function:
+
+```ocaml
+val load_dir : dir:Path.t -> Loaded.t Memo.t
+```
+
+`Loaded.t` represents a "loaded" set of rules for a particular
+directory. It can also be thought as a "compiled" set of rules. A
+`Loaded.t` contains all the rules in existence that produce targets in
+`dir`. For instance, given a `Loaded.t` we can figure out all the
+files that would be produced in `dir` if we were building everything
+that could be built. While `dir` is a build directory, this also
+includes files present in the source tree. This is because
+`Load_rules.load_dir` implicitly adds copy rules for all source files
+present in the source directory that correspond to the build directory
+`dir`. For instance, if `dir` is `_build/default/src`, Dune will
+implicitly add rules to copy files in `src` to `_build/default/src`.
+Except for rules that have the special `promote` or `fallback` modes.
+
+This is in fact how Dune evaluates globs during the build. Indeed,
+when writing `dune` files we work in an imaginary world where both the
+source files and the generated files are present. So when we write
+`(deps (glob_files *.txt))`, this `*.txt` denotes both `.txt` files
+that are present on disk in the source tree but also as the ones that
+can be generated by the build.
+
+In practice, to evaluate `(glob_files *.txt)` in directory `d`, Dune
+calls `Load_rules.load_dir ~dir:d` and filter the list of files that can
+be built. Similarly, when Dune needs to build a file
+`_build/default/src/x`, it first calls `Load_rules.load_dir` with
+`_build/default/src` and then looks up a rule that has `x` has
+target in the returned `Loaded.t`. The `Load_rules.load_dir` is
+memoised, so it can be called multiple times during the build without
+guilt.
+
+While `Load_rules` is responsible for driving the production of rules,
+it is part of `dune_engine` which doesn't know about `dune` files and
+doesn't know about OCaml libraries or OCaml compilation in general. So
+it is not responsible for actually producing the build rules that
+allow to build Dune projects. Instead, `Load_rules` defers the actual
+production of rules to a callback that it obtains via
+`Build_config`. Inside Dune, this callback is implemented by the
+`Gen_rules` module inside the `dune_rules` library. `dune_rules` is
+the library that is responsible for parsing, interpreting and
+compiling `dune` files down to low-level build rules.
+
+### `Dune_rules.Gen_rules`
+
+The entry of `Dune_rules.Gen_rules` is the `gen_rules` function. Its
+API looks like:
+
+```ocaml
+val gen_rules :
+  Build_config.Context_or_install.t ->
+  dir:Path.Build.t ->
+  string list ->
+  Build_config.gen_rules_result Memo.t
+```
+
+Where `Build_config.gen_rules_result` is, in most cases —when the value
+returned is `Build_config.Rules _`—, a "raw" set of rules. Raw in the sense
+that there is no overlap checks or any other checks. During the rule
+production phase, we merely accumulate a set of rules that is later
+processed. The API of `gen_rules` is in fact a bit more complex, but
+the above definition is enough for the purpose of this document.
+
+The first thing `gen_rules` does is analyse the directory it is
+given. If the directory corresponds to a source directory with a `dune`
+file, `gen_rules` will dispatch the call to the part of `dune_rules`
+that parses and interprets the `dune` file. This is the simplest case,
+but even in this case there are some things worth mentioning.
+
+For instance, when compiling an OCaml library dune stores the
+artifacts for the library in generated dot-directories. For instance,
+the cmi files for library `foo` living in source directory `src` will
+end up in `_build/default/src/.foo.objs/byte`. We could produce these
+rules when `gen_rules` is called with directory
+`_build/default/src/.foo.objs/byte`, however that would spread out the
+logic for interpreting `library` stanzas. It is much simpler to
+produce all the build rules corresponding to a `library` stanza in one
+go. This is what is happening at the moment: when called with
+directory `_build/default/src`, `gen_rules` will not only produce
+rules for this directory but will also produce rules for
+`_build/default/src/.foo.objs/byte` and various other directories.
+
+`Load_rules` doesn't know anything about this. And in particular, it
+doesn't know that it is the `gen_rules` call for directory
+`_build/default/src/` that will produce the rules for the dot
+subdirectories. When `Load_rules` loads the rules for the
+`.../.foo.objs/byte` sub-directory, it simply calls `gen_rules` with
+this directory. It is `gen_rules` that "redirects" the call to the
+`_build/default/src` directory by calling
+`Load_rules.load_dir_and_produce_its_rules`. This function simply
+calls `Load_rules.load_dir` and re-emits all the raw rules that were
+returned by the corresponding `gen_rules` call.
+
+This works because `Load_rules.load_dir` accepts the facts that
+`gen_rules` produces rules for many directory at once. It simply
+filters out the result. But for things to behave well, the unwritten
+following invariant must hold: `gen_rules ~dir:d` is allowed to
+generate rules for directory `d'` iff `gen_rules ~dir:d'` emits a call
+to `Load_rules.load_dir ~dir:d`.
+
+This scenario happens in a number of cases. All these cases share a
+common pattern: the redirections are always to an ancestor
+directory. At the moment, there is one exception to this pattern in
+the odoc rules, however it is easy to remove.
+
+Finally, the `copy_files` stanza creates another form of dependency
+between directory. In order to calculate the targets produced by
+`copy_files`, which needs to be known at rule production time, we need
+to evaluate the glob given to `copy_files`. Which requires doing a
+call to `Load_rules.load_dir` as previously described. Contrary to the
+other form of dependency we just describe, this ones can go from any
+directory to any other directory. For instance, the following stanza
+in `src/dune`:
+
+```
+(copy_files foo/*.txt)
+```
+
+would create a dependency from `_build/default/src` to
+ `_build_default/src/foo`.
+
+So in the end, if we were looking at the internal computation graph of
+Dune and narrowing it to just the calls to `Load_rules.load_dir`, we
+would see a graph with many edges going from a directory to one of its
+ancestor. These would mostly be between generated dot-subdirectories
+and their first ancestor that has a corresponding directory in the
+source tree. Plus a few other arbitrary ones for each `copy_files`
+stanza.
+
+## Directory targets
+
+Before directory targets, answering the question "what rules produces
+file X?" was easy. Dune would just call `Load_rules.load_dir` and
+lookup `X` in the result. With directory targets, things are a bit
+more complicated. Indeed, `X` might also be produced by a directory
+target in an ancestor directory. This means that `Load_rules.load_dir`
+now need to look in parent directories as well, which introduce more
+dependencies from directories to their parents and can create cycles
+because of `copy_files` stanza that create dependencies in the other
+direction.
+
+At a result, some combinations of `copy_files` and directory targets
+don't produce the expected result. This is documented in the test
+suite.