unikernel/duniverse/dune_/doc/dev/rule-streaming.md

@@ -0,0 +1,82 @@
+# Rule streaming
+
+This document describes a new design for the production of build rules
+in Dune. The new design aims to be more natural, easier to reason
+about and to make existing features work well with newer ones such as
+directory targets.
+
+It was originally written by Jérémie Dimino as part of the
+[streaming RFC](https://github.com/ocaml/dune/pull/5251), and later on moved
+into the dev documentation.
+
+
+## Problem
+
+The [rule production](./rule-production.md) document exposes a concrete problem
+with directory targets, but there is also a general sense of messiness in the
+way things work. Generating rules for multiple directories at once is
+natural, but the current encoding is odd.
+
+## Proposal
+
+The proposal is to add the following rule: `gen_rules ~dir` is allowed
+to produced rules in `dir` or any of its descendant only. It is not
+allowed to produce rules anywhere else.
+
+`Load_rules.load_dir ~dir` will then always call itself recursively on
+the parent of `dir` and take the union of the rules produced by
+`gen_rules` for `dir` and the ones produced by the recursive
+call. `gen_rules` will no longer have to redirect a call via
+`Load_rules.load_dir_and_produce_its_rules`, which we would simply
+remove.
+
+This introduces a cycle with all `copy_files` stanza that copy files
+from a sub-directory. We propose the break this cycle by introducing
+laziness in the rule production code.
+
+### Generating rules with a mask
+
+The idea is that when we produce rules, we will produce rules under
+a current active "mask" that tells us where we are allowed to generate
+files or directories.  Trying to produce a rule with targets not
+matched by this mask will be a runtime error.
+
+When entering `gen_rules ~dir`, the initial mask will be: "any files
+and directories that is a descendant of directory `dir`".
+
+We can then narrow the mask to a sub-mask:
+
+```ocaml
+val narrow : Target_mask.t -> unit Memo.t -> unit Memo.t
+```
+
+With `narrow mask m`, `m` would only be allowed to produce rules whose
+target are matched by the intersection of `mask` and the current
+mask. `m` wouldn't be evaluated eagerly. Instead, `gen_rules` would
+now return a set of direct rules as well as a list of
+`(Target_mask.t * unit Memo.t)`. Let's call such a pair a
+suspension. A suspension can be forced by evaluation its second
+component. Doing so will yield a list of rules matched by the mask and
+a new list of suspension.
+
+### Staged rules loading
+
+The next step is to stage `Load_rules.load_dir`. In addition to taking
+a directory, `load_dir` will now also take a mask and will return the
+set of rules for this mask. To do that, it might need to force a bunch of
+suspensions recursively.
+
+
+### How does that help?
+
+We will put `copy_rules` under a `narrow <only file targets in current
+dir>`. In order to determine if a directory is part of a directory
+target in an ancestor directory, we wouldn't need to force this
+suspension.
+
+### Difficulties
+
+Interpreting a `library` stanza requires knowing the set of `.ml`
+files in the current directory. Knowing this requires interpreting
+`copy_files` in the current directory. So the interpretation of
+`library` stanzas will need to go under a `narrow` as well.