3.1 KiB
Rule streaming
This document describes a new design for the production of build rules in Dune. The new design aims to be more natural, easier to reason about and to make existing features work well with newer ones such as directory targets.
It was originally written by Jérémie Dimino as part of the streaming RFC, and later on moved into the dev documentation.
Problem
The rule production document exposes a concrete problem with directory targets, but there is also a general sense of messiness in the way things work. Generating rules for multiple directories at once is natural, but the current encoding is odd.
Proposal
The proposal is to add the following rule: gen_rules ~dir is allowed
to produced rules in dir or any of its descendant only. It is not
allowed to produce rules anywhere else.
Load_rules.load_dir ~dir will then always call itself recursively on
the parent of dir and take the union of the rules produced by
gen_rules for dir and the ones produced by the recursive
call. gen_rules will no longer have to redirect a call via
Load_rules.load_dir_and_produce_its_rules, which we would simply
remove.
This introduces a cycle with all copy_files stanza that copy files
from a sub-directory. We propose the break this cycle by introducing
laziness in the rule production code.
Generating rules with a mask
The idea is that when we produce rules, we will produce rules under a current active "mask" that tells us where we are allowed to generate files or directories. Trying to produce a rule with targets not matched by this mask will be a runtime error.
When entering gen_rules ~dir, the initial mask will be: "any files
and directories that is a descendant of directory dir".
We can then narrow the mask to a sub-mask:
val narrow : Target_mask.t -> unit Memo.t -> unit Memo.t
With narrow mask m, m would only be allowed to produce rules whose
target are matched by the intersection of mask and the current
mask. m wouldn't be evaluated eagerly. Instead, gen_rules would
now return a set of direct rules as well as a list of
(Target_mask.t * unit Memo.t). Let's call such a pair a
suspension. A suspension can be forced by evaluation its second
component. Doing so will yield a list of rules matched by the mask and
a new list of suspension.
Staged rules loading
The next step is to stage Load_rules.load_dir. In addition to taking
a directory, load_dir will now also take a mask and will return the
set of rules for this mask. To do that, it might need to force a bunch of
suspensions recursively.
How does that help?
We will put copy_rules under a narrow <only file targets in current dir>. In order to determine if a directory is part of a directory
target in an ancestor directory, we wouldn't need to force this
suspension.
Difficulties
Interpreting a library stanza requires knowing the set of .ml
files in the current directory. Knowing this requires interpreting
copy_files in the current directory. So the interpretation of
library stanzas will need to go under a narrow as well.