83 lines
3.1 KiB
Markdown
83 lines
3.1 KiB
Markdown
|
# Rule streaming
|
||
|
|
|
||
|
|
This document describes a new design for the production of build rules
|
||
|
|
in Dune. The new design aims to be more natural, easier to reason
|
||
|
|
about and to make existing features work well with newer ones such as
|
||
|
|
directory targets.
|
||
|
|
|
||
|
|
It was originally written by Jérémie Dimino as part of the
|
||
|
|
[streaming RFC](https://github.com/ocaml/dune/pull/5251), and later on moved
|
||
|
|
into the dev documentation.
|
||
|
|
|
||
|
|
|
||
|
|
## Problem
|
||
|
|
|
||
|
|
The [rule production](./rule-production.md) document exposes a concrete problem
|
||
|
|
with directory targets, but there is also a general sense of messiness in the
|
||
|
|
way things work. Generating rules for multiple directories at once is
|
||
|
|
natural, but the current encoding is odd.
|
||
|
|
|
||
|
|
## Proposal
|
||
|
|
|
||
|
|
The proposal is to add the following rule: `gen_rules ~dir` is allowed
|
||
|
|
to produced rules in `dir` or any of its descendant only. It is not
|
||
|
|
allowed to produce rules anywhere else.
|
||
|
|
|
||
|
|
`Load_rules.load_dir ~dir` will then always call itself recursively on
|
||
|
|
the parent of `dir` and take the union of the rules produced by
|
||
|
|
`gen_rules` for `dir` and the ones produced by the recursive
|
||
|
|
call. `gen_rules` will no longer have to redirect a call via
|
||
|
|
`Load_rules.load_dir_and_produce_its_rules`, which we would simply
|
||
|
|
remove.
|
||
|
|
|
||
|
|
This introduces a cycle with all `copy_files` stanza that copy files
|
||
|
|
from a sub-directory. We propose the break this cycle by introducing
|
||
|
|
laziness in the rule production code.
|
||
|
|
|
||
|
|
### Generating rules with a mask
|
||
|
|
|
||
|
|
The idea is that when we produce rules, we will produce rules under
|
||
|
|
a current active "mask" that tells us where we are allowed to generate
|
||
|
|
files or directories. Trying to produce a rule with targets not
|
||
|
|
matched by this mask will be a runtime error.
|
||
|
|
|
||
|
|
When entering `gen_rules ~dir`, the initial mask will be: "any files
|
||
|
|
and directories that is a descendant of directory `dir`".
|
||
|
|
|
||
|
|
We can then narrow the mask to a sub-mask:
|
||
|
|
|
||
|
|
```ocaml
|
||
|
|
val narrow : Target_mask.t -> unit Memo.t -> unit Memo.t
|
||
|
|
```
|
||
|
|
|
||
|
|
With `narrow mask m`, `m` would only be allowed to produce rules whose
|
||
|
|
target are matched by the intersection of `mask` and the current
|
||
|
|
mask. `m` wouldn't be evaluated eagerly. Instead, `gen_rules` would
|
||
|
|
now return a set of direct rules as well as a list of
|
||
|
|
`(Target_mask.t * unit Memo.t)`. Let's call such a pair a
|
||
|
|
suspension. A suspension can be forced by evaluation its second
|
||
|
|
component. Doing so will yield a list of rules matched by the mask and
|
||
|
|
a new list of suspension.
|
||
|
|
|
||
|
|
### Staged rules loading
|
||
|
|
|
||
|
|
The next step is to stage `Load_rules.load_dir`. In addition to taking
|
||
|
|
a directory, `load_dir` will now also take a mask and will return the
|
||
|
|
set of rules for this mask. To do that, it might need to force a bunch of
|
||
|
|
suspensions recursively.
|
||
|
|
|
||
|
|
|
||
|
|
### How does that help?
|
||
|
|
|
||
|
|
We will put `copy_rules` under a `narrow <only file targets in current
|
||
|
|
dir>`. In order to determine if a directory is part of a directory
|
||
|
|
target in an ancestor directory, we wouldn't need to force this
|
||
|
|
suspension.
|
||
|
|
|
||
|
|
### Difficulties
|
||
|
|
|
||
|
|
Interpreting a `library` stanza requires knowing the set of `.ml`
|
||
|
|
files in the current directory. Knowing this requires interpreting
|
||
|
|
`copy_files` in the current directory. So the interpretation of
|
||
|
|
`library` stanzas will need to go under a `narrow` as well.
|