swrup/mte
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Browse source
This commit is contained in:
swrup 2025-11-11 02:07:51 +01:00
parent aa2ff7b2f0
commit 2f3113f55d
11742 changed files with 1223940 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

5
unikernel/duniverse/dune_/doc/dev/README.md Normal file
View file

@ -0,0 +1,5 @@
# Developer documentation
We use `doc/dev` for storing developer documentation, including specification,
design and implementation notes. Anything that should be user-facing should go
into the `doc` folder instead.

279
unikernel/duniverse/dune_/doc/dev/cache.md Normal file
View file

@ -0,0 +1,279 @@
# Dune cache: design and implementation notes
This document describes main ideas behind the Dune cache as well as a few
subtleties of the implementation. This is a working document and it will be
updated as part of on-going development. Some of the described features are
currently still in development or unused, in particular, here we describe
support for two types of cache entries – artifacts and values – but
Dune currently doesn't store any value entries in the cache.
The design and implementation are based on a non-trivial assumption that there
are no hash collisions, for instance, that we will never come across two files
with different contents but with the same content hash. While this assumption is
common in the world of build systems and package managers, and is highly
unlikely to be violated by chance, one can manufacture hash collisions on
purpose, especially when using a weak hash like MD5.
## What we store in the cache
The cache stores build _artifacts_ and _values_.
* An _artifact_ is a file produced by a build rule. As any file, it has a name
as well as content. Note that we treat a file's executable permission bit as
part of its content.
* A _value_ is anything else produced during a build that is not written to a
file but which is worth storing persistently between successive builds. A
common example is a string written to the standard output by a build action.
Such output-producing actions may run as part of a build rule or at an earlier
stage when generating rules. Unlike artifacts, values have no names, yet they
still have content.
## How we use the cache
The build system uses the cache to _store_ and _restore_ build artifacts and
values. Here is a typical interaction sequence for the case of artifacts:
* The build system is executing a build rule and has already identified all of
its dependencies, thereby obtaining the _rule hash_. It uses it to make a
_restore request_ to the cache.
* Now there are two cases:
- The cache successfully restores all the artifacts of the rule, placing
them into the build directory and returning their content hashes. The
build system can skip building the artifacts and can use the obtained
content hashes for deciding whether any dependent build rules need to be
rerun. **This successful scenario is the only reason we use the cache.**
- The cache fails to restore the artifacts, either because of an error or
because it doesn't have an entry for the given rule hash. In this case,
the build system needs to build the artifacts itself. On completion, it
makes a _store request_ to the cache, providing a list of build artifacts
(_file names_ in the build directory) as well as their _content hashes_.
The cache stores the artifacts and after that the build system is allowed
to continue with the build (but not before, since the cache requires
exclusive access to the artifacts in the build directory). There can be a
rare situation where the store request is declined because the given entry
is already in the cache. How could this happen? We did try to restore it
first! The reason is that the cache can be populated concurrently by
multiple build systems and by the distributed cache daemon, so there can
be a race between multiple systems adding the same entry to the cache. In
this case, the cache will perform _deduplication_ of build artifacts by
replacing them with hard links to the copies already stored in the cache
(this is an example where the exclusive access is needed).
Build values are handled similarly; the main difference is that to identify a
cache entry we use the corresponding _action hash_ rather than the _rule hash_.
The only difference between rule hashes and action hashes is that the former
include the names of the produced artifacts into the hash, while the latter
do not, since values have no names.
## Cache storage format
Let `root` stand for the cache root directory. It has three main subdirectories.
* `root/meta/v3` stores _metadata files_, one per each historically executed
build rule or value-producing action. (While this is a convenient mental
model, in reality we need to occasionally remove some outdated metadata files
to free disk space – see the section on cache trimming.)
<br/><br/>
A metadata file corresponding to a build rule is named by the rule hash and
stores file names and content hashes of all artifacts produced by the rule.
<br/><br/>
A metadata file corresponding to a value-producing action is named by the
action hash and stores the hash of the resulting value.
<br/><br/>
It is important to guarantee that rule and action hashes do not accidentally
overlap, which may happen if one simply hashes their in-memory representations
because a rule and an action might happen to be represented by the same
sequence of bytes in memory.
* `root/files/v3` is a storage for artifacts, where files named by content
hashes store the matching contents. We will create hard links to these files
from build directories and rely on the hard link count, as well as on the last
change time as useful metrics during cache trimming.
* `root/values/v3` is a storage for values. As in the case of `files`, we store
the values in the files named by their content hashes. However, these files
will always have the hard link count equal to 1, because they do not appear
anywhere in build directories. By storing them in a separate directory, we
simplify the job of the cache trimmer.
* `root/temp` contains temporary files used for atomic file operations needed
when adding new entries to the cache, as will be described below.
Note that since this document was first written, some of the above paths have
changed due to version bumps (to `v4` and beyond).
## Adding entries to the cache
To add entries to the cache, we use the functions `store_artifacts` and
`store_value` described in the corresponding sections below. Setting possible
errors aside, these functions can succeed in two ways.
* They return `Stored` if the given entry is new and it has been successfully
stored in the cache.
* They return `Already_present` if the given entry has already been present in
the cache and can therefore be discarded. This is a rare scenario where
multiple systems race to add the same entry, and only one of them will receive
the glory of the `Stored` response.
### Atomic writing to the cache
As mentioned above, the cache can be modified concurrently by multiple systems,
so to prevent collisions on individual files, we need to create new files
atomically. To do that, we first create a temporary file in the `temp`
directory, then create a hard link to it from the cache (this operation will
fail if another process managed to create the cache entry earlier), and then
unlink the temporary file.
From now on, whenever we say "create a file", we mean create a file atomically.
If two systems attempt to create a file with the same name simultaneously, one
of them will win the competition and the contents it writes will remain in the
cache until it is deleted during cache trimming.
Note that it is possible for a metadata file with a given name to have multiple
possible contents due to _non-determinism_, and the cache implementation should
not assume otherwise.
### Storing artifacts
To store artifacts produced by a build rule, we perform the following sequence
of steps.
* Create a metadata file in the `meta` directory, listing all the artifacts.
If the file already exists (which should be a rare case), verify that it
contains the expected list of artifacts (both file names and content hashes).
If it doesn't, we have found a non-deterministic build rule and report an
error.
* For each artifact, we store it in the `files` directory using the artifact's
content hash as the name. In each case, there are two scenarios:
- If the artifact is already in the cache, we perform
deduplication by replacing the artifact in the build directory
with a hard link to the file stored in the cache. We assume that
the build system will wait for `store_artifacts` to complete before
starting any further actions that might read these artifacts from
the build directory and thus interfere with the deduplication.
- Otherwise, we create a hard link to the artifact from the `files`
directory.
The function returns `Already_present` if the metadata file and all of the
artifacts were already in the cache; otherwise, it returns `Stored`.
### Storing values
Storing a value is simpler than storing artifacts because there is no need
for deduplication. The steps are:
* Create a metadata file in the `meta` directory, recording the value's hash.
If the file already exists (which should be a rare case), verify that it
contains the same hash. If it doesn't, we have found a non-deterministic build
action and report an error.
* Store the value as a file in the `values` directory using the value's hash as
the file name. If the file is already in the cache, we don't need to do
anything.
The function returns `Already_present` if the metadata file and the value were
already in the cache; otherwise, it returns `Stored`.
## Restoring entries from the cache
To restore entries from the cache, we use the functions `restore_artifacts` and
`restore_value` described in the corresponding sections below. Setting possible
errors aside, these functions either fail to find the entry in the cache and
return `Not_found_in_cache`, or succeed and return `Restored` along with some
information about the restored entry.
### Restoring artifacts
Given a rule hash, the function `restore_artifacts` performs the following
steps.
* Look up the corresponding metadata file in the `meta` directory. If it doesn't
exist, return `Not_found_in_cache`. Otherwise, read the list of artifacts,
i.e. the list of file names and their content hashes from the metadata file.
* For each artifact, lookup the content hash in the `files` directory. If it
doesn't exist, return `Not_found_in_cache`. Otherwise: (i) delete the
corresponding (most likely stale) file in the build directory, and then (ii)
create a hard link with the same name, pointing to the file in the cache.
If the above succeeds for every artifact in the list, the function returns
`Restored` along with the obtained list of file name and content hash pairs.
### Restoring values
Given an action hash, the function `restore_value` performs the following steps.
* Look up the corresponding metadata file in the `meta` directory. If it doesn't
exist, return `Not_found_in_cache`. Otherwise, read the hash of the value.
* Look up the hash in the `values` directory. If it doesn't exist, return
`Not_found_in_cache`. Otherwise, return `Restored` along with the value read
from the stored file.
## Trimming the cache
Storing all historically produced artifacts and values is infeasible, so the
cache needs to be regularly trimmed. The current trimming algorithm performs the
following steps.
* Scan the `files` directory to find all currently unused artifact entries. An
artifact is _unused_ if its hard link count is equal to 1. There is no point in
trimming other entries, since they appear in at least one build directory. In
fact, trimming them is potentially harmful because if the same entries were to
be added to the cache again from a new directory, we would have been unable to
perform the deduplication, thus losing some sharing opportunities.
* Scan the `values` directory to find all value entries. We have no information
about their current usage, so we conservatively allow all of them to be
trimmed and recomputed in the next build if needed.
* Sort the entries according to the following criteria:
- Type: artifacts precede values in the trimming list since artifacts are
generally larger and we know for sure that they are unused;
- The time of last change: entries that became unused more recently go later
in the list.
* Traverse the list and delete the corresponding entries until the trimming goal
has been met. Right before deleting an artifact entry, double check that its
hard link count is still equal to 1. A build system running concurrently might
have created a hard link to it after we collected the information, so deleting
this file from the cache could lead to a loss of sharing between different
build directories.
* Finally, traverse the `meta` directory and remove all _broken_ metadata files,
i.e. the files that refer to content hashes with no corresponding entries in
the `files` and `values` directories. This step does not need to be done on
every trimming. It is expensive but metadata files are generally small and
there is no harm in keeping broken metadata files in the cache. In fact, the
information contained in broken metadata files can be utilised by the build
system for so-called _shallow builds_ where intermediate build artifacts are
not materialised on the disk and it is sufficient to only know their hashes,
which are listed in the metadata files.
To enable more sophisticated trimming strategies, we could augment the metadata
stored in the cache with information about the _cost_ of producing cache
entries, i.e. the time it would take to execute the corresponding rule or action
to restore the entry if needed. For deterministic build rules, we can do _local
cost reasoning_, i.e. we do not need to take the cost of rebuilding their
dependents into account, since such rebuilding would be unnecessary due to the
early cut-off optimisation.
Another promising idea is to add support for incremental cache trimming where
the build system informs the cache that a previously added entry has become
obsolete, letting the cache trim it early if it meets the trimming criteria.
### Interaction with the previous cache versions
Note also that as the new cache format evolves further and we, for example, move
from `files/v3` to `files/v4`, the cache trimmer will need to evolve too, to be
able to cope with entries of all currently supported versions.

64
unikernel/duniverse/dune_/doc/dev/directory-targets.md Normal file
View file

@ -0,0 +1,64 @@
# Directory targets
A **directory target** corresponds to a file tree rooted at a specified root
directory, for example, `docs/*`. Typical examples of rules producing directory
targets are: unpacking an archive, running `make` in a vendored package, and
building files with non-deterministic names (e.g. including the current date).
## Declaring a directory target
To declare a directory target `docs/*`, use the syntax `(target (dir docs))` in
a rule stanza. The corresponding rule should create the directory `docs` and is
free to populate it with an arbitrary number of files and/or subdirectories.
Like file targets, directory targets can be promoted to the source tree by
adding `(mode promote)` to the rule stanza.
## Depending on a directory target
There are two ways to depend on a directory target:
* An **opaque dependency** on the whole file tree `docs/*`. Opaque dependencies
are invalidated if the contents of the tree is changed in any way. To declare
an opaque dependency on `docs/*`, use the syntax `(dep (dir docs))` in a rule
stanza.
* A **projection dependency** on a specific file in the tree, e.g.
`docs/html/index.html`. A projection dependency is declared using the standard
syntax `(dep docs/html/index.html)` and works like a dependency on a normal
file target. For example, if the `docs/*` directory is rebuilt and only
`docs/html/logo.png` is modified, then the dependency on `docs/html/index.html`
is considered to be up-to-date. Note that it is easy to make a mistake with
such projection dependencies, for example, by forgetting that `index.html`
actually does include the image `docs/html/logo.png`. In such cases,
sandboxing will help since only the requested projection dependencies will be
available in the sandbox (i.e., not the whole directory target).
## Building a directory target
Users can request building whole directory targets or individual files via
`dune build docs` and `dune build docs/html/index.html` commands.
## Current limitations
* It is not allowed to have two rules with the same directory target. That is,
like file targets, directory targets are **exclusive** (but see _shared
directory targets_ below).
* Directory targets cannot have nested file or directory targets, i.e. other
rules are not allowed to declare targets within the file tree of a directory
target.
## Possible future extensions
Here are some possible extensions to consider:
* **Opaque directory targets**: a rule may declare that its directory target is
opaque, in which case projection dependencies on its content will be
disallowed. One can also consider only partially opaque directory targets,
where the contents of the directory is only partially visible.
* **Shared directory targets**: we can allow multiple rules to write to the same
directory target, as long as they do not write to the same files. In this
case, depending on a directory target would mean depending on all of the rules
that declare it as a target.

106
unikernel/duniverse/dune_/doc/dev/rev-store.md Normal file
View file

@ -0,0 +1,106 @@
The Revision Store
==================
The revision store is the place where Git data that is relevant to the Dune
package management is cached.
The Concepts
------------
The revision store uses Git in the way of its original slogan, as a
content-addressable file system. A lot of data (code) and meta-data (opam files)
is stored in Git repositories that are often forked from each other, hence to
save space Dune has a Git object cache.
Git is implemented as a way to store revisions and being able to address them.
However, these revisions do not have to have a common ancestor and given Git
uses SHA1 hashes for addressing it is possible to join multiple repositories in
one single Git repository without clashing. A fairly common usecase outside of
Dune are `gh-pages` branches to serve documentation on Github, which do not
share a common parent with the main branch of the repository.
The revision store exploits this feature by putting all revisions of all
repositories into one single large repository to take advantage of caching
effects.
The Advantages
--------------
This way of organizing means that all revisions shared between multiple forks
of a Git repository can reuse the same Git objects that are in common and don't
need to download them nor store them again. Updating repositories can be done
incrementally, as Git knows which revisions are available locally and which
ones need to be fetched.
It is also possible to refer to previous states easily as the commits are part
of a Git repo and checking out an older version is as simple as checking out
the current version of the files.
Considerations and Compromises
------------------------------
An important consideration was that the management of the revision store should
be entirely transparent to the user, they should not need to do any steps to
create nor maintain it. It should get created automatically if needed and all
the steps that are necessary to keep it updated should happen in the
background. The store should always work like a cache that can be discarded
safely without causing data loss.
The revision store should always give out the most recent version of data,
unless explicitly instructed otherwise. This means that :
* If only a Git source is specified, then the revision store will
automatically get the newest revision
* If the source specifies a tag or branch, then the revision store will
automatically update to the newest revision
* It a revision is specified, the revision store will only update if the
revision is not yet cached, otherwise the cached version can be used
The final consideration means that an offline usage is possible if all
repositories specified are specified with their hash.
Due to the fact that the revision store is a Git repository it means that the
data sources that can be added to it also have to be available via Git. This
means that adding repositories that use different version control systems
aren't supported at the moment nor are plain HTTP sources supported.
Support for other kinds of VCSes is a possible extension by replicating similar
concepts with other version control systems, provided they allow for similar
flexibility as the Git way of storing revisions. However at the moment most
users have settled on using Git, hence this version should be able to
accommodate the needs for most users.
Another compromise is that old repositories with long histories and large sizes
have to be cloned before use, thus increasing the size of the initial download
compared to the same metadata downloaded as a compressed tarball. Despite Git
compressing objects, the history of the repositories to be added does increase
the overhead.
A solution to this could be shallow clones which only contain the latest
revisions, however these have [shown to be
problematic](https://blog.cocoapods.org/Master-Spec-Repo-Rate-Limiting-Post-Mortem/)
thus for time being we are fetching the complete histories.
Implementation
--------------
This section describes the current way the revision store is implemented.
As the revision store is not project specific, it is stored in the user's cache
directory (using the [freedesktop.org](https://www.freedesktop.org/wiki/)
specifications, the directory specified by `XDG_CACHE_HOME`), with all dune
instances sharing one single revision store.
The revision store itself is a `bare` Git repository without a worktree. This
is because all repositories in the revision store are equal and checking out
one particular revision would be a waste of disk space as the Git tooling can
be used to construct any revisions out of the bare repository anyway.
Thus every source that is added to the revision store as a remote that tracks
the default branch (or, if a branch is specified explicitly, then that
branch) and fetched, thus storing the required revisions in the revision store.
The implementation of these features is a mix of calling the `git` binary and
implementing parts in OCaml. This means that the `git` binary is required on
the system. Possible future improvements could be using
[ocaml-git](https://github.com/mirage/ocaml-git) to avoid the dependency.

180
unikernel/duniverse/dune_/doc/dev/rpc-versioning.md Normal file
View file

@ -0,0 +1,180 @@
# Runtime RPC versioning implementation notes
This document describes the versioning protocol used to ensure two-way
compatibility between different versions of the API.
The approach is loosely inspired by the `Both_converts` model used by
[Versioned_rpc](https://ocaml.janestreet.com/ocaml-core/latest/doc/async_rpc_kernel/Async_rpc_kernel/index.html#module-Versioned_rpc)
in `Async`, in which both parties maintain a "menu" of supported RPC
versions, which is used to negotiate a common protocol for each
method.
This is a working document and will be updated as the design evolves.
## Terms
- A **procedure** is a common term encompassing *notifications*
(one-way messages) and *requests* (a communication to which
a response is expected).
- A **model** is the logical payload type for one direction of
a procedure. Note that this is a per-actor entity; the ultimate
goal of runtime RPC versioning is to allow clients and servers to
disagree on a model type without preventing them from interacting.
- A **wire type** for a procedure is the logical type sent "over the
wire" for one direction of a procedure. The end result of
negotiating a version for a procedure is to select a wire type
known to both the client and the server. Typically, the older of
the two model types will be chosen as the wire type,
- A **generation** of a procedure is the set of wire types
corresponding to each direction of a procedure, along with the
de/serialisation logic and upgrade and downgrade functions
transforming the wire types to the model types and vice versa.
Each generation is associated with a *version number*, which
should be unique within a procedure.
- The **menu** is a mapping from method names to the particular
generations that will be used for each procedure for a particular
session. In the source code, this term is overloaded to also refer
to a mapping from method names to *all known* generations of
a procedure.
- The **declaration** of a procedure lists its model types and all
known generations, along with its method name. Multiple
declarations of the same procedure is allowed, so long as they do
not overlap version numbers.
- The **implementation** of a declaration is the actual behavior of
a procedure, which acts on the model types. Typically, this will
be on the server, but in the future there may also be a use for
server-to-client requests. This document is not concerned with the
internals of any given implementation, only whether such an
implementation exists at all.
## Background
Previously, there was no distinction between model and wire types.
This meant that any change to a model type required both build servers
and clients to upgrade in lockstep, as otherwise the receiver would be
unable to deserialize the payload of a procedure.
Unfortunately, most lighter-weight solutions (such as modifying the
de/serialization logic to be resilient to, e.g., extra/missing fields
or variants in types) are insufficient. Early designs of the
diagnostic API, for example, reported targets as strings, but was
changed to give structured information instead.
Similarly, requirements like "the client must always be older than the
server" (or the reverse) don't work in environments like Jane Street,
where the same editor plugin must be able to interact seamlessly with
multiple iterations of Dune (which may be older or newer than the
editor plugin itself).
The main goal of the system, then, is to ensure that both the server
and client applications can be programmed against the current model
types for each procedure, with all backwards- or forwards- conversions
happening under the hood.
## Protocol
At session initialization time, the client will first send an
initialization request to the server containing a single version
number corresponding to the overall RPC version the client will use.
If this number is determined to be versioning-compatible (see
[Session versioning](#session-versioning)), the server will respond
with a token instructing the client to initiate version negotiation.
Otherwise, the server will respond with an error.
Upon receiving this token, the client will initiate version
negotiation by sending a list of `(method-name, generations)`
pairs, where `method-name` is the name of each declared procedure, and
`generations` is the list of version numbers for that procedure's
generations.
Upon receiving a list of supported versions from the client, the
server will compare it to its list of *implemented* versions,
selecting the greatest common generation for each procedure. If the
client and server do not share any generations for a procedure, it is
omitted entirely. If there is at least one method for which a common
version exists, then the server responds with a list of `(method-name,
selected-version)` pairs, where `selected-version` is the version
number of the greatest common generation. This list is then used by
both parties to construct the version menu. Otherwise, if there are no
common versions for any methods, an error is returned to the client
and the session is invalidated.
Note that we do not currently require declared/implemented versions to
span a contiguous range of version numbers. This can have a few uses,
such as preventing clients from using a known-bugged generation of
a procedure.
When executing a procedure, the sender first looks up the correct
generation in the menu (see [Error handling](#error-handling)), and
downgrades the payload from the sender-side model type to the wire
type. Upon receipt, the server performs the same lookup to deserialize,
then upgrade the payload to the receiver-side model type, then the
procedure implementation is performed, producing a response in the
case of requests. If necessary, the same transformations are then
performed in reverse, sending the value back to the sender, completing
the procedure.
Barring strange circumstances (such as a client declaring a generation
with a newer version number than the type exposed in `dune_rpc.mli`), it
is always the case that the transformation from wire to model types will
be the identity function on the side that is older.
## Miscellaneous implementation notes
### Session versioning
In addition to version numbers existing for each procedure version,
there are two further version numbers associated with the session as
a whole which are sent as part of session initialization.
The first is the version of Dune each side purports to be as
a `MAJOR.MINOR` number (serialized as an `int * int` pair). This is
not currently checked.
Next is a version of the initial handshake protocol to be used. This
takes the form of a single `int`. In the future, if the initial
negotiation protocol changes, this value can be adjusted and checked to
account for this.
### Error handling
Handling of versioning errors has become more complex, as we need to
distinguish between "no such method exists" and "the server and client
do not share any common generations for this method". Secondly, this
means that the initiation of a procedure can now fail, which
complicates one-way communications (for example, the server must
swallow errors and clients must be upgraded to handle version errors
on notifications, which were previously infallible).
Finally, the versioning protocol itself must be either versioned
separately or stabilised (see [Session versioning](#session-versioning)).
### Tweaks
- We currently send the entire version menu from client to server and
back twice, once for the client to inform the server of all
supported versions, and again for the server to inform the client
of the common versions. This can lead to large messages being
passed at session initialization, which may become a performance
bottleneck.
- The size of the version negotiation messages is proportional to
the number of all known generations for all procedures, which
can be approximated by `number-of-procedures` times
`number-of-supported-generations`. In practice, I do not
expect this number to be large (I would be surprised if this
number is ever on the order of 100).
- One alternative is to perform per-procedure negotiation, where
the initiator of a procedure first sends its known version
ranges, the recipient sends the selected version (or an
error), then the procedure proceeds as before. This approach
trades startup and lookup overhead for a constant
per-communication overhead. It also makes distinguishing "no
such method exists" and "no common versions" simpler.

176
unikernel/duniverse/dune_/doc/dev/rule-production.md Normal file
View file

@ -0,0 +1,176 @@
# Rule production
This document describes how rule production works in Dune. It was originally
written by Jérémie Dimino as part of the
[streaming RFC](https://github.com/ocaml/dune/pull/5251), but moved
into the dev documentation as it provides a great overview on how this part of
Dune works at present.
## How does rule production works?
### `Dune_engine.Load_rules`
The production of rules is driven by the module `Load_rules` in the
`dune_engine` library. This library is the build system core of
Dune. It is meant as a general purpose library for writing build
systems, and the Dune software is built on top of it. In theory,
`dune_engine` shouldn't know about `dune` or `dune-project`
files. However, for historical reason this is not the case yet and
`dune_engine` still knows some things about them.
As we work on Dune, we expect that `dune_engine` will become more and
more agnostic. Even though it is not completely agnostic, we have
successfully been using it to build Jane Street code base, using the
Jane Street rules on top of this core. So it's already more general
than Dune itself.
For the purpose of this design doc, we will treat `dune_engine` as a
completely general library that doesn't know about `dune` files.
The main feature of `Load_rules` is the `Load_rules.load_dir` function:
```ocaml
val load_dir : dir:Path.t -> Loaded.t Memo.t
```
`Loaded.t` represents a "loaded" set of rules for a particular
directory. It can also be thought as a "compiled" set of rules. A
`Loaded.t` contains all the rules in existence that produce targets in
`dir`. For instance, given a `Loaded.t` we can figure out all the
files that would be produced in `dir` if we were building everything
that could be built. While `dir` is a build directory, this also
includes files present in the source tree. This is because
`Load_rules.load_dir` implicitly adds copy rules for all source files
present in the source directory that correspond to the build directory
`dir`. For instance, if `dir` is `_build/default/src`, Dune will
implicitly add rules to copy files in `src` to `_build/default/src`.
Except for rules that have the special `promote` or `fallback` modes.
This is in fact how Dune evaluates globs during the build. Indeed,
when writing `dune` files we work in an imaginary world where both the
source files and the generated files are present. So when we write
`(deps (glob_files *.txt))`, this `*.txt` denotes both `.txt` files
that are present on disk in the source tree but also as the ones that
can be generated by the build.
In practice, to evaluate `(glob_files *.txt)` in directory `d`, Dune
calls `Load_rules.load_dir ~dir:d` and filter the list of files that can
be built. Similarly, when Dune needs to build a file
`_build/default/src/x`, it first calls `Load_rules.load_dir` with
`_build/default/src` and then looks up a rule that has `x` has
target in the returned `Loaded.t`. The `Load_rules.load_dir` is
memoised, so it can be called multiple times during the build without
guilt.
While `Load_rules` is responsible for driving the production of rules,
it is part of `dune_engine` which doesn't know about `dune` files and
doesn't know about OCaml libraries or OCaml compilation in general. So
it is not responsible for actually producing the build rules that
allow to build Dune projects. Instead, `Load_rules` defers the actual
production of rules to a callback that it obtains via
`Build_config`. Inside Dune, this callback is implemented by the
`Gen_rules` module inside the `dune_rules` library. `dune_rules` is
the library that is responsible for parsing, interpreting and
compiling `dune` files down to low-level build rules.
### `Dune_rules.Gen_rules`
The entry of `Dune_rules.Gen_rules` is the `gen_rules` function. Its
API looks like:
```ocaml
val gen_rules :
Build_config.Context_or_install.t ->
dir:Path.Build.t ->
string list ->
Build_config.gen_rules_result Memo.t
```
Where `Build_config.gen_rules_result` is, in most cases —when the value
returned is `Build_config.Rules _`—, a "raw" set of rules. Raw in the sense
that there is no overlap checks or any other checks. During the rule
production phase, we merely accumulate a set of rules that is later
processed. The API of `gen_rules` is in fact a bit more complex, but
the above definition is enough for the purpose of this document.
The first thing `gen_rules` does is analyse the directory it is
given. If the directory corresponds to a source directory with a `dune`
file, `gen_rules` will dispatch the call to the part of `dune_rules`
that parses and interprets the `dune` file. This is the simplest case,
but even in this case there are some things worth mentioning.
For instance, when compiling an OCaml library dune stores the
artifacts for the library in generated dot-directories. For instance,
the cmi files for library `foo` living in source directory `src` will
end up in `_build/default/src/.foo.objs/byte`. We could produce these
rules when `gen_rules` is called with directory
`_build/default/src/.foo.objs/byte`, however that would spread out the
logic for interpreting `library` stanzas. It is much simpler to
produce all the build rules corresponding to a `library` stanza in one
go. This is what is happening at the moment: when called with
directory `_build/default/src`, `gen_rules` will not only produce
rules for this directory but will also produce rules for
`_build/default/src/.foo.objs/byte` and various other directories.
`Load_rules` doesn't know anything about this. And in particular, it
doesn't know that it is the `gen_rules` call for directory
`_build/default/src/` that will produce the rules for the dot
subdirectories. When `Load_rules` loads the rules for the
`.../.foo.objs/byte` sub-directory, it simply calls `gen_rules` with
this directory. It is `gen_rules` that "redirects" the call to the
`_build/default/src` directory by calling
`Load_rules.load_dir_and_produce_its_rules`. This function simply
calls `Load_rules.load_dir` and re-emits all the raw rules that were
returned by the corresponding `gen_rules` call.
This works because `Load_rules.load_dir` accepts the facts that
`gen_rules` produces rules for many directory at once. It simply
filters out the result. But for things to behave well, the unwritten
following invariant must hold: `gen_rules ~dir:d` is allowed to
generate rules for directory `d'` iff `gen_rules ~dir:d'` emits a call
to `Load_rules.load_dir ~dir:d`.
This scenario happens in a number of cases. All these cases share a
common pattern: the redirections are always to an ancestor
directory. At the moment, there is one exception to this pattern in
the odoc rules, however it is easy to remove.
Finally, the `copy_files` stanza creates another form of dependency
between directory. In order to calculate the targets produced by
`copy_files`, which needs to be known at rule production time, we need
to evaluate the glob given to `copy_files`. Which requires doing a
call to `Load_rules.load_dir` as previously described. Contrary to the
other form of dependency we just describe, this ones can go from any
directory to any other directory. For instance, the following stanza
in `src/dune`:
```
(copy_files foo/*.txt)
```
would create a dependency from `_build/default/src` to
`_build_default/src/foo`.
So in the end, if we were looking at the internal computation graph of
Dune and narrowing it to just the calls to `Load_rules.load_dir`, we
would see a graph with many edges going from a directory to one of its
ancestor. These would mostly be between generated dot-subdirectories
and their first ancestor that has a corresponding directory in the
source tree. Plus a few other arbitrary ones for each `copy_files`
stanza.
## Directory targets
Before directory targets, answering the question "what rules produces
file X?" was easy. Dune would just call `Load_rules.load_dir` and
lookup `X` in the result. With directory targets, things are a bit
more complicated. Indeed, `X` might also be produced by a directory
target in an ancestor directory. This means that `Load_rules.load_dir`
now need to look in parent directories as well, which introduce more
dependencies from directories to their parents and can create cycles
because of `copy_files` stanza that create dependencies in the other
direction.
At a result, some combinations of `copy_files` and directory targets
don't produce the expected result. This is documented in the test
suite.

82
unikernel/duniverse/dune_/doc/dev/rule-streaming.md Normal file
View file

@ -0,0 +1,82 @@
# Rule streaming
This document describes a new design for the production of build rules
in Dune. The new design aims to be more natural, easier to reason
about and to make existing features work well with newer ones such as
directory targets.
It was originally written by Jérémie Dimino as part of the
[streaming RFC](https://github.com/ocaml/dune/pull/5251), and later on moved
into the dev documentation.
## Problem
The [rule production](./rule-production.md) document exposes a concrete problem
with directory targets, but there is also a general sense of messiness in the
way things work. Generating rules for multiple directories at once is
natural, but the current encoding is odd.
## Proposal
The proposal is to add the following rule: `gen_rules ~dir` is allowed
to produced rules in `dir` or any of its descendant only. It is not
allowed to produce rules anywhere else.
`Load_rules.load_dir ~dir` will then always call itself recursively on
the parent of `dir` and take the union of the rules produced by
`gen_rules` for `dir` and the ones produced by the recursive
call. `gen_rules` will no longer have to redirect a call via
`Load_rules.load_dir_and_produce_its_rules`, which we would simply
remove.
This introduces a cycle with all `copy_files` stanza that copy files
from a sub-directory. We propose the break this cycle by introducing
laziness in the rule production code.
### Generating rules with a mask
The idea is that when we produce rules, we will produce rules under
a current active "mask" that tells us where we are allowed to generate
files or directories. Trying to produce a rule with targets not
matched by this mask will be a runtime error.
When entering `gen_rules ~dir`, the initial mask will be: "any files
and directories that is a descendant of directory `dir`".
We can then narrow the mask to a sub-mask:
```ocaml
val narrow : Target_mask.t -> unit Memo.t -> unit Memo.t
```
With `narrow mask m`, `m` would only be allowed to produce rules whose
target are matched by the intersection of `mask` and the current
mask. `m` wouldn't be evaluated eagerly. Instead, `gen_rules` would
now return a set of direct rules as well as a list of
`(Target_mask.t * unit Memo.t)`. Let's call such a pair a
suspension. A suspension can be forced by evaluation its second
component. Doing so will yield a list of rules matched by the mask and
a new list of suspension.
### Staged rules loading
The next step is to stage `Load_rules.load_dir`. In addition to taking
a directory, `load_dir` will now also take a mask and will return the
set of rules for this mask. To do that, it might need to force a bunch of
suspensions recursively.
### How does that help?
We will put `copy_rules` under a `narrow <only file targets in current
dir>`. In order to determine if a directory is part of a directory
target in an ancestor directory, we wouldn't need to force this
suspension.
### Difficulties
Interpreting a `library` stanza requires knowing the set of `.ml`
files in the current directory. Knowing this requires interpreting
`copy_files` in the current directory. So the interpretation of
`library` stanzas will need to go under a `narrow` as well.