unikernel/duniverse/dune_/doc/explanation/bootstrap.rst

@@ -0,0 +1,53 @@
+How Dune Uses Dune to Build Dune
+================================
+
+Dune's build system is itself Dune. This works thanks to a bootstrap process.
+This document explains how this works.
+
+``boot/bootstrap.ml``
+---------------------
+
+``boot/bootstrap.ml`` is an OCaml script (it is interpreted, not compiled) that
+is a mini-build system tailored to Dune itself. It computes dependencies
+between the various modules by calling ``ocamldep``, and it will generate build
+and link commands. It knows how to execute these commands in parallel. It does
+not read any ``dune`` file. However, the project structure and its system
+dependencies are encoded in ``boot/libs.ml``.
+
+This step produces ``_boot/dune.exe``.
+
+Completing the Opam Installation
+--------------------------------
+
+``_boot/dune.exe`` is the bootstrap Dune. Since it has been built from
+the Dune sources, it will act like Dune: it can read ``dune`` files, etc.
+
+This is actually the ``dune`` executable that will get installed. This is the
+same executable as the one obtained by running ``opam install dune``. At this
+stage of the process, Opam does not know about this: it expects a
+``dune.install`` file that explains what files to install.
+
+The next command run by the Opam instruction is the following:
+
+.. code:: console
+
+   $ ./_boot/dune.exe build dune.install --release --profile dune-bootstrap
+
+By using the ``dune-bootstrap`` :term:`build profile`, it does not run a full
+build, but only copy ``_boot/dune.exe`` to its install location (as the `dune`
+binary), and generate ``dune.install``.
+
+``make dev``: Everything Else for Local Development
+---------------------------------------------------
+
+The above describes how Dune itself is built through Opam, but that's not all
+there is it to it: the Dune repository contains other libraries that need to be
+built. In fact, executing the ``boot/bootstrap.ml`` script did not generate
+files useful for editor integration.
+
+So the main ``Makefile`` has a ``make dev`` target that will run
+``_boot/dune.exe build @install``: this will rebuild the project using Dune
+itself.
+
+As a special rule, this build will regenerate ``boot/libs.ml`` using the
+locations of the internal libraries used to build Dune.

unikernel/duniverse/dune_/doc/explanation/index.rst

@@ -0,0 +1,17 @@
+Explanation
+===========
+
+These documents explain how certain feature works, or how Dune integrates with
+the rest of the OCaml ecosystem.
+
+.. toctree::
+   :maxdepth: 1
+
+   scopes
+   preprocessing
+   ocaml-ecosystem
+   package-management
+   opam-integration
+   bootstrap
+   mental-model
+   tour/index

unikernel/duniverse/dune_/doc/explanation/mental-model.rst

@@ -0,0 +1,202 @@
+The Dune Mental Model
+=====================
+
+It is not strictly necessary to understand Dune's underlying model to use it;
+but knowing how it works under the hood will help writing build rules, and also
+help understand some errors and what's possible with Dune.
+
+.. note::
+
+   This document is a simplification of the reality: the actual rules might be
+   different, it does not touch rule loading and glosses over how caching
+   works, but should be a useful tool to build an understanding of Dune.
+
+How Dune Works
+--------------
+
+The building block of Dune is the *rule*:
+
+    A *rule* reads *dependencies* and writes *targets* using an *action* (and
+    it can be attached to *aliases*).
+
+When ``dune build`` is executed, it will first read the project's ``dune``
+files to determine the rules that apply to the project. Once it has done this,
+it will determine what actions it needs to execute to build the required
+targets.
+
+An Example
+----------
+
+Let's take the following example.
+
+- there's a CLI tool written in OCaml.
+- it has some build-time configuration stored in ``config.json``.
+- it has an integration test, in which the tool is executed with
+  ``testdata.txt`` as input.
+
+Configuration Generation
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+To express the generation of the configuration module we could write:
+
+.. code:: dune
+
+  (rule
+   (deps convert/json2ml.exe config.json)
+   (target config.ml)
+   (action
+    (run convert/json2ml.exe config.json -o config.ml)))
+
+This rule will:
+
+- read its dependencies: ``convert/json2ml.exe`` and ``config.json``
+- and write its target: ``config.ml``
+- using an action: ``(run convert/json2ml.exe config.json -o config.ml)``
+
+This rule is very explicit: we write a stanza for a single Dune rule.
+
+Building the Executable
+^^^^^^^^^^^^^^^^^^^^^^^
+
+In contrast, to describe the compilation of the executable, we would write:
+
+.. code:: dune
+
+  (executable
+   (name tool)
+   (modules main config))
+
+Here, we use Dune's abstractions. Dune knows about the OCaml compilation model:
+the modules need to be compiled and linked together. So it will generate the
+following rules under the hood:
+
+- one rule to compile the ``Main`` module:
+
+  - it will read its dependency: ``main.ml``
+  - and write its output: ``main.cmx``
+  - using an action: ``(run ocamlopt -c main.ml)``
+
+- one rule to compile the ``Config`` module:
+
+  - it will read its dependency: ``config.ml``
+  - and write its output: ``config.cmx``
+  - using an action: ``(run ocamlopt -c config.ml)``
+
+- one rule to link the ``tool.exe`` executable:
+
+  - it will read its dependencies: ``main.cmx`` and ``config.cmx``
+  - and write its output: ``tool.exe``
+  - using an action: ``(run ocamlopt -o tool.exe main.cmx config.cmx``)
+
+Note that in this example, some files are targets of a rule and dependencies of
+another (``.cmx`` files). We are unlikely to ever interact with them directly,
+so it can also be useful to think of the ``(executable)`` stanza as a group of
+rules with ``main.ml`` and ``config.ml`` as inputs and ``tool.exe`` as output.
+
+Running the Tests
+^^^^^^^^^^^^^^^^^
+
+Some rules do not produce any output file, but we're still interested in
+running their actions. A test is a good example: we want the build process to
+exit with an error code if the action fails. In that case, the rule does not
+have targets, but we "attach" it to an :term:`alias`, ``runtest`` in this case.
+This gives us a way of requesting this rule to be executed. As we are about to
+see, rules are executed lazily by asking for their targets to be built, so we
+would not be able to execute such rules.
+
+.. code:: dune
+
+  (rule
+   (deps tool.exe testdata.txt)
+   (alias runtest)
+   (action
+    (run tool.exe testdata.txt)))
+
+This rule:
+
+- reads its dependencies: ``tool.exe`` and ``testdata.txt``
+- writes no targets
+- using an action: ``(run tool.exe testdata.txt)``
+- (and it is attached to ``runtest``)
+
+What to Build
+-------------
+
+Dune can build *files* and *aliases*. These can be found on the command line:
+
+- ``dune build tool.exe`` will build the ``tool.exe`` file.
+- ``dune build @example`` will build the ``example`` alias.
+- ``dune build tool.exe @example`` will build both the file ``tool.exe`` and
+  the ``example`` alias.
+- ``dune runtest`` is a shortcut for ``dune build @runtest``: it will build the
+  ``runtest`` alias. Passing a directory will build all tests in that directory.
+  Passing the path to a cram test will run that test individually.
+- ``dune build`` is a shortcut for ``dune build @@default``: it will build the
+  default alias in the current directory (by default the ``all`` alias).
+
+In other words, each ``dune build`` or ``dune runtest`` command always
+corresponds to a list of files and aliases to build.
+
+.. seealso:: :doc:`Reference information on aliases</reference/aliases>`
+
+How Dune Interprets Rules
+-------------------------
+
+We have now seen that Dune sets up rules for a project, and that every build
+command has a list of files and aliases that we are asking to build.
+
+Now let's see how this request is processed:
+
+- to build a file, Dune will first check if it is in the source tree. In that
+  case, there is nothing to do. Otherwise, it will check if it is the
+  target of a rule. In that case, it will execute this rule. (Dune will raise
+  an error in other cases: if the file is both in the source tree and the
+  target of a rule, or if it is neither)
+- to build an alias, Dune will execute all the rules that are attached to this
+  alias.
+- to execute a rule, Dune will first build all the dependencies (files or
+  aliases) of this rule. Then it will execute the action attached to the rule.
+  When Dune is about to execute an action, it checks (in various caches) if it
+  executed it before on the same set of dependencies, and, if yes, it can skip
+  executing it and reuse the previous result.
+
+In the case of our example, if we call ``dune runtest``, Dune will consider all
+rules attached to the ``runtest`` alias. In this case it is just the
+integration test rule. It needs to build its dependencies, ``tool.exe`` and
+``testdata.txt``. The latter is present in the source tree.
+However, ``tool.exe`` is the target of the linking rule defined by the
+``(executable)`` stanza. This rule requires ``main.cmx`` and ``config.cmx``.
+``main.cmx`` is the target of the compilation rule for the ``Main`` module,
+which depends on ``main.ml``. This file is in the source tree, so let's copy it
+under ``_build``. This rule has all its dependencies available, so we can run
+its action, which writes ``main.cmx``. Getting back to the dependencies of
+``tool.exe``, ``config.cmx`` is the target of the linking rule of the
+``Config`` module. This rule has ``config.ml`` has a dependency. This file is
+itself the target of the configuration module rule, which lists ``config.json``
+and ``convert/json2ml.exe``. The first is available in the source tree and to
+simplify, let's assume that the second one has been built. This action has all
+its dependencies available, so we can execute its action to produce its target,
+``config.ml``. Now the module compilation rule for ``Config`` can be executed,
+producing ``config.cmx``; and in turn the linking rule can be executed,
+producing ``tool.exe``. Finally, ``tool.exe`` can be executed with
+``testdata.txt`` as its argument.
+
+In a nutshell: we recursively copied all the dependencies of the test rule, and
+executed the rules in the correct order.
+
+This is a "cold build", where there were no previous build artifacts. Note that
+if we change only part of the project (say the ``main.ml`` file), only a small
+number of rules will be evaluated, the ones that depend on ``main.ml``.
+
+Conclusion
+----------
+
+Dune's underlying model is based on rules. Stanzas are high-level constructs
+that can generate multiple rules, that are not always visible.
+
+To build a target, Dune looks for the rule that produces that target and makes
+its way back to source files.
+
+Rules define a directed acyclic graph which models dependency relations between
+files. Most of the rules in that graph may be executed for a cold build, but
+just the minimum will be executed for an incremental build.

unikernel/duniverse/dune_/doc/explanation/ocaml-ecosystem.rst

@@ -0,0 +1,94 @@
+The OCaml Ecosystem
+===================
+
+The OCaml ecosystem is not monolithic: the compiler and tools are not
+maintained by the same entities. As such, it can be difficult to understand the
+history and roles of the various pieces of this ecosystem. The goal of this
+page is to give a quick overview of the situation and the role that Dune
+plays in it.
+
+The OCaml Compiler Distribution: Compiling and Linking
+------------------------------------------------------
+
+The `OCaml compiler distribution <https://github.com/ocaml/ocaml>`_ contains
+"core" tools including the compilers (``ocamlc`` and ``ocamlopt``). They turn
+source files (with extensions ``.ml`` and ``.mli``) into executables and
+libraries. Dependencies between compiled objects only exist at the module
+level, so this is a low-level tool.
+
+Findlib: Metadata for Libraries
+-------------------------------
+
+Findlib_ is a tool that defines the concept of library, so that libraries can
+depend on other libraries on top of the notion of module. Definitions of
+libraries, and other pieces of metadata, are stored in ``META`` files.
+
+Findlib ships an executable named ``ocamlfind`` that can be used as a wrapper
+on top of the compilers to perform tasks such as producing an executable from
+compiled object files and external libraries.
+
+.. _findlib: https://github.com/ocaml/ocamlfind
+
+Opam: a Collection of Software Projects
+---------------------------------------
+
+Opam is a package manager. It is used to determine which packages are
+necessary, and how to fetch and build them. Packages can contain libraries,
+executables, and other kinds of files.
+
+The notion of version is specific to opam. If your project uses a function
+named ``Png.read_file`` but this function has been added only in version
+``1.2.0`` of that package, opam needs to know about it.
+
+Opam manages collections of installed packages, called switches. Using your
+project's dependencies (names and version constraints), it is able to create a
+switch that you'll be using to develop your project.
+
+Public definitions of packages are available in a database called
+``opam-repository`` which is maintained as a public Git repository. Publishing a
+package on opam (to make sure that external users can use your project)
+consists in adding its definition to ``opam-repository``.
+
+Dune: Giving Structure to Your Source Tree
+------------------------------------------
+
+Dune is a build system. It is used to orchestrate the compilation of source
+files into executables and libraries.
+
+Assuming you have a development switch set up, you communicate to Dune about how your
+project is organized in terms of executables, libraries, and tests. It is then able to assemble the source files of your projects, with the dependencies installed in an opam switch, to create compiled assets for your project.
+
+How Dune Integrates With the Ecosystem
+--------------------------------------
+
+Dune is designed to integrate with the tools mentioned above:
+
+- By knowing how the OCaml compilers operate, it knows which build commands should be
+  re-executed if some source files change.
+- It outputs metadata like dependency information into ``META`` files that
+  Findlib is able to make use of. This ensures that even if a project does not use Dune, it
+  can use a library that has been produced by Dune. Conversely, it can read
+  these files to determine dependency information for dependencies that have
+  not been produced by Dune.
+- It is able to generate opam files with filenames consistent with how opam
+  looks for them. The generated files use build commands that make use of the
+  :doc:`/reference/aliases/install` and ``@runtest`` :term:`aliases <alias>` so
+  that the Dune abstractions map to the opam ones.
+
+Dune is Opinionated
+-------------------
+
+As described above, the OCaml ecosystem does not have a centralized toolchain.
+Units such as modules, libraries, and packages operate at different levels, and
+the relation between these can be confusing to users.
+
+Dune tries to simplify the picture by reducing the difference between these
+objects:
+
+- By default, a library will only expose a single top-level module named after
+  the library (this is called a wrapped library).
+- A library can only be installed in the package of the same name. This means
+  that the names found in ``dune-project`` and ``opam`` files (package names)
+  are consistent with the names found in ``dune`` files (library names). More
+  precisely, libraries ``foo``, ``foo.bar`` and ``foo.baz`` are part of the
+  ``foo`` package.

unikernel/duniverse/dune_/doc/explanation/opam-integration.rst

@@ -0,0 +1,119 @@
+How Dune integrates with opam
+=============================
+
+.. highlight:: opam
+
+When instructed to do so (see :doc:`../howto/opam-file-generation`), Dune generates opam files with the following instructions::
+
+  build: [
+    ["dune" "subst"] {dev}
+    [
+      "dune"
+      "build"
+      "-p"
+      name
+      "-j"
+      jobs
+      "@install"
+      "@runtest" {with-test}
+      "@doc" {with-doc}
+    ]
+  ]
+
+Let's see what this means in detail.
+
+Substitution
+------------
+
+The first step is to call ``dune subst``, but only if the ``{dev}`` opam
+variable is set. This variable is only set when the package is pinned.
+This means that :ref:`dune-subst` does not run for released versions, but it
+does for development versions.
+
+This is not a problem since released versions should have a ``(version)`` field
+set in ``dune-project``, and :term:`placeholder substitution` should have been
+performed. `dune-release`_ takes care of these steps.
+
+.. _dune-release: https://github.com/tarides/dune-release
+
+Opam Variables
+--------------
+
+In the second command line, ``name`` is a variable that evaluates to the name
+of the package being built, and ``jobs`` is a variable that corresponds to the
+number of commands to run in parallel.
+
+What ``-p`` Means
+-----------------
+
+The ``-p`` flag, shorthand for ``--release-of-packages``, is Dune's public interface to set up the options for an opam build. The exact semantics may change, but as of Dune 3.8 it is equivalent to the combination of:
+
+- ``--root .``: set the :term:`root` to prevent Dune from :ref:`looking it up <finding-root>`.
+- ``--only-packages name``: ignore packages other than ``name`` defined in the project.
+- ``--profile release``: set the :term:`build profile` to ``release``. In particular, this ensures that warnings are not fatal.
+- ``--ignore-promoted-rules``: silently ignores all rules with ``(mode promote)``.
+- ``--default-target @install``: make sure that ``dune build`` with no target argument builds ``@install``, not ``@@default`` (this is not used in the opam integration since an explicit target is passed)
+- ``--no-config``: do not load the configuration file in the user's home directory.
+- ``--always-show-command-line``: ensures that the programs executed by Dune end up in the opam logs.
+- ``--promote-install-files``: ensures that ``*.install`` files are present in the source tree after the build.
+- ``--require-dune-project-file``: fail if ``dune-project`` is not present. In some previous Dune versions, ``dune-project`` could be generated when it is not present. This is not desirable with opam since the version the package has been prepared with is not known.
+
+The Targets We're Building
+--------------------------
+
+The targets are specified as::
+
+  "@install"
+  "@runtest" {with-test}
+  "@doc" {with-doc}
+
+The ``{with- }`` syntax is an opam filter. It means that the string before is
+present or not depending on the opam variable. These variables, in turn, are set depending on the opam configuration.
+
+Concretely, in the next table, if the opam command on the left is executed, the Dune target on the right will be built:
+
+.. list-table::
+   :header-rows: 1
+
+   * - opam command
+     - Dune target
+   * - ``opam install pkg``
+     - ``@install``
+   * - ``opam install pkg --with-test``
+     - ``@install @runtest``
+   * - ``opam install pkg --with-test --with-doc``
+     - ``@install @runtest @doc``
+
+This filtering mechanism is also used to declare dependencies.
+If a package is using ``lwt`` and ``alcotest``, but the latter only in its test
+suite, its ``depends:`` field is::
+
+  "lwt"
+  "alcotest" {with-test}
+
+This is expanded to just ``"lwt"`` in ``opam install pkg``, but to ``"lwt"
+"alcotest"`` in ``opam install pkg --with-test``.
+
+The meaning of these :term:`aliases <alias>` is the following:
+
+- :doc:`/reference/aliases/install` depends on all the ``*.install`` files in
+  the project. In turn, these depend on all the installable files (libraries and
+  executables with a public name and files that are manually installed through
+  ``(install)`` stanzas).
+- :doc:`/reference/aliases/runtest` is the alias to which all tests are
+  attached, including ``(test)`` stanzas. ``dune build @runtest`` is equivalent
+  to ``dune runtest``.
+- :doc:`/reference/aliases/doc` executes ``odoc`` to create HTML docs under
+  ``_build``.
+
+What Opam Expects From Dune
+---------------------------
+
+Given this ``build:`` lines and the fact that there is no ``install:`` line,
+what happens is the following:
+
+- Opam executes ``dune subst``, if the package is being pinned.
+- Opam executes the build instruction, usually just ``dune build -p pkg @install``
+- This Dune command builds all the installable files and creates a ``pkg.install`` file.
+- This file contains the paths to built files (somewhere in the ``_build`` directory) and the opam sections they should be installed in.
+- Opam interprets this file and copies the built files to their destination. The install file is also used as a manifest of which files belong to which package, which is used when uninstalling the package.

unikernel/duniverse/dune_/doc/explanation/package-management.md

@@ -0,0 +1,234 @@
+# How Package Management Works
+
+This document explains how Dune's package management works under the hood. It
+requires a bit of familiarity with how opam repositories work and how Dune
+builds packages. Thus it is aimed at people who want to understand how the
+feature works, not how it is used.
+
+For a tour on how to apply package management to a project, refer to the
+{doc}`/tutorials/dune-package-management/index` tutorial.
+
+## Motivation
+
+A core part of modern programming is using existing code to save time. The
+OCaml package ecosystem has quite a long history with many projects building
+upon each other over many years. A significant step forward was the creation of
+the OCaml Package Manager, opam, along with the establishment of a public
+package repository which made it a lot more feasible to share code between
+people and projects.
+
+Over time, best practices have evolved, and while opam has incorporated some
+changes, it couldn't adopt all the modern workflows due to its existing user
+base and constraints.
+
+Thus the Dune Package Management has been designed with a few core goals in
+mind:
+
+* No global state visible to users, everything is local to projects
+* Package management is configured through files (`dune-project` and optionally
+  `dune-workspace`)
+* Repositories are automatically kept up to date unless explicitly configured
+  to use specific versions
+* Builds can only access packages they have declared dependencies on
+* Reproducible builds through lockfiles
+
+Dune plays well with the existing OCaml ecosystem and does not introduce a new
+type of packages. Rather, it uses the same package repository and Dune packages
+stay installable with opam.
+
+## Package Management in a Project
+
+This section describes what happens in a Dune project using the package
+management feature.
+
+## Dependency Selection
+
+The first step is to determine which packages need to be installed.
+Traditionally this has been defined in the `depends` field of a project's opam
+file(s).
+
+Since version 1.10 Dune has supported {doc}`opam file generation
+</howto/opam-file-generation>` by specifying the package dependencies in the
+`dune-project`.
+
+The package management feature uses the same metadata, as Dune will determine
+the list of packages to install from the `depends` field in the `dune-project`
+file. This allows projects to completely omit generation of `.opam` files, as
+long as they use Dune for package management. Thus all dependencies on OCaml
+packages are only declared in one single file.
+
+To maintain compatibility with a large number of existing projects, Dune
+continues to support `.opam` files. While it is recommended to declare the
+dependencies directly in the `dune-project` file, it is not mandatory to do so.
+Dune will fall back to reading dependencies from `.opam` files when the package
+is not defined in `dune-project`.
+
+
+## Locking
+
+Given the list of the project's dependencies and their version
+constraints, the next steps are:
+
+1. Find the transitive dependencies and figure out a version for each
+   dependency that satisfies the constraints
+2. For each dependency, download it, build it, and make it available to the
+   project
+
+In opam, `opam install` does both of these.
+
+In Dune, these are separate steps: the first one is `dune pkg lock`, and the
+second one happens implicitly as part of [building](#building).
+
+The idea of doing the first step and recording it for later is popular in other
+programming language package managers like NPM and is usually called locking.
+Creating a lock file ensures that the dependencies to be installed are
+always the same - unless that lock file is updated of course.
+
+:::{note}
+`opam` also supports creating lock files. However, these are not as central to
+the opam workflow as they are in the case of package management in Dune, which
+always requires a set of locked packages.
+:::
+
+In the most general sense, a lock file is just a set of specific packages
+and their versions to be installed.
+
+Instead of a lock file, Dune writes this information to a directory (the "lock
+directory") with files that describe the dependencies. It includes the
+package's name and version. Unlike many other package managers, the files
+include a lot of other information as well, such as the location of the source
+archives to download (since there is no central location for all archives), the
+build instructions (since each package can use its own way of building), and
+additional metadata like the system packages it depends upon.
+
+The information is stored in a directory (`dune.lock` by default) as separate
+files, to reduce potential merge conflicts and simplify code review. Storing
+additional files like patches is also simpler this way.
+
+### Package Repository Management
+
+To find a valid solution that allows a project to be built, it is necessary to
+know what packages exist, what versions of these packages exist, and what other
+packages these depend on, etc.
+
+In opam, this information is tracked in a central repository called
+[`opam-repository`](https://github.com/ocaml/opam-repository), which contains
+all the metadata for published packages.
+
+It is managed using Git; opam typically uses a snapshot to find the
+dependencies when searching for a solution that satisfies the constraints.
+
+Likewise, Dune uses the same repository; however, instead of snapshots of the
+contents, it uses the Git repository directly.
+
+:::{note}
+Dune maintains a shared internal cache containing all Git repositories that
+projects use. This way updates and checkouts are very fast because only new
+revisions have to be retrieved. The downside is that to be included in the
+cache, all the Git repos have to be cloned first which depending on the size of
+the repositories can take a bit of time.
+:::
+
+On every call to `dune pkg lock`, Dune will update the metadata repository
+first (hence why efficiently updating that repository matters). This means that
+each `dune pkg lock` will use the newest set of packages available.
+
+However, it is also possible to declare specific revisions of the repositories,
+to get a reproducible solution. Due to using Git, any previous revision of the
+repository can be used by specifying a commit hash.
+
+Dune uses two repositories by default:
+
+* `upstream` refers to the default branch of `opam-repository`, which contains
+  all the publicly released packages.
+* `overlay` refers to
+  [opam-overlay](https://github.com/ocaml-dune/opam-overlays), which defines
+  packages patched to work with package management. The long-term goal is to
+  have as few packages as possible in this repository as more and more packages
+  work within Dune Package Management upstream. Check the
+  [compatibility](#compatibility) section for details.
+
+### Solving
+
+After Dune has read the constraints and loaded set of candidate packages, it is
+necessary to determine which packages and versions should be selected for the
+package lock.
+
+To do so, Dune uses
+[`opam-0install-solver`](https://github.com/ocaml-opam/opam-0install-solver),
+which is a variant of the [`0install`](https://github.com/0install/0install)
+solver to find solutions for opam packages.
+
+Contrary to opam, the Dune solver always starts from a blank slate; it assumes
+nothing is installed and everything needs to be installed. This has the
+advantage that solving is now simpler, and previous solver solutions don't
+interfere with the current one. Thus, given the same inputs, it should always
+come up with the same result; no state is held between the solver runs.
+
+This can lead to more packages being installed (as opam won't install new
+package versions by default if the existing versions satisfy the constraints),
+but it avoids interference from already installed packages that lead to
+potentially different solutions.
+
+After solving is done, the solution gets written into the lock directory with
+all the metadata necessary to build and install the packages. From this point
+on, there is no need to access the package metadata repositories.
+
+:::{note}
+Solving and locking does not download the package sources. These are downloaded
+in the build step.
+:::
+
+(building)=
+## Building
+
+When building, Dune will read the information from the lock directory and set
+up rules for the packages. Check {doc}`/explanation/mental-model` for details
+about rules.
+
+The rules that the package management sets up include:
+
+* Fetch rules to download and unpack the source archives, and also download any
+additional sources such as patches
+* Build rules to execute the build instructions stored in the lock directory
+* Install rules to put the artifacts that were built into the appropriate
+  Dune-managed folders
+
+Creating these processes as rules mean that they will only be executed on
+demand, so if the project has already downloaded the sources, it does not need
+to download them again. Likewise, if packages are installed, they stay
+installed.
+
+The results of the rules are stored in the project's `_build` directory and
+managed automatically by Dune. Thus, when cleaning the build directory, the
+installed packages are cleaned as well and will be reinstalled at the next
+build.
+
+(compatibility)=
+## Packaging for Dune Compatibility
+
+Dune can build and install most packages as dependencies, even if they are not
+built with Dune themselves. Dune will execute the build instructions from the
+lock directory, very similar to opam.
+
+However, packages must adhere to certain rules to be compatible with Dune.
+
+The most important one is that the packages must not use absolute paths to
+refer to files. That means they cannot read the path they are being built or
+installed in and expect this path to remain the same. Dune builds packages in a
+sandbox location, and after the build has finished, it moves the files to the
+actual destination.
+
+:::{note}
+Unlike opam, Dune at the moment does not wrap the build in sandboxing tools
+like [Bubblewrap](https://github.com/containers/bubblewrap).
+:::
+
+To comply with these restrictions the usual solution is to use relative paths,
+as Dune guarantees that packages installed into different sections are
+installed in a way where their relative location stays the same.
+
+The `overlay` repository exists specifically to make currently non-compliant
+packages compatible with Dune's package management. It does so by supplying
+releases of packages where the current upstream releases don't support Dune
+package management yet.

unikernel/duniverse/dune_/doc/explanation/preprocessing.rst

@@ -0,0 +1,55 @@
+How Preprocessing Works
+=======================
+
+Preprocessing consists in transforming source code before it is compiled. The
+goal of this document is to explain how this works in Dune.
+
+Dune supports two separate ways of applying preprocessors, the "classic pipeline" (used
+with ``(staged_pps)``), and the "fast pipeline" (used for all other
+:doc:`preprocessing specifications <../reference/preprocessing-spec>` including
+``(pps)``).
+
+The OCaml compilers provide options for specifying a preprocessing step. The
+``-pp`` option is used to invoke a textual preprocessor (something that reads
+text and returns text). The ``-ppx`` option is used to invoke a `ppx rewriter`
+(a function that takes an AST and outputs an AST).
+
+This is the "classic pipeline": preprocessing is part of the compilation
+itself. This is simple, but has a problem: in order to compute the dependencies
+of a module, it is necessary to pass the same ``-pp`` or ``-ppx`` option to
+``ocamldep``.
+
+The classic pipeline has the following steps:
+
+- preprocessing (as part of ``ocamldep``)
+- dependency analysis
+- preprocessing (as part of compilation)
+- compilation
+
+Dune supports a "fast pipeline" where the preprocessor is invoked separately
+from the compiler and its output is saved. Afterwards the preprocessed code is
+compiled directly.
+
+The fast pipeline has the following steps:
+
+- preprocessing
+- dependency analysis
+- compilation
+
+It has several advantages: it only invokes the preprocessor once per file, and
+the preprocessed code is reused between dependency analysis and different kinds
+of compilation. Also, when several preprocessors use ``ppxlib``, they can be
+combined in a preprocessing program that traverses the AST only once.
+
+However, some specific code generators or preprocessors require direct
+access to the compilation artefacts of their dependencies. Therefore they
+need to be used with the classic pipeline, even if it is slower. Note that a
+PPX is able to know if it was called as part of ``ocamldep -ppx`` or ``ocamlopt
+-ppx``, so it can act differently in each phase.
+
+Dune chooses which pipeline to use depending on the
+provided :doc:`../reference/preprocessing-spec`. It will select the fast pipeline,
+unless ``(staged_pps)`` is used. In that case, the classic pipeline is used.
+
+In the case of the fast pipeline, a single executable is built and accepts
+arguments for all preprocessors.

unikernel/duniverse/dune_/doc/explanation/scopes.rst

@@ -0,0 +1,43 @@
+Dune Projects and Workspaces
+============================
+
+Whenever Dune builds anything, it does so at the level of a *Dune workspace*. A
+Dune workspace is a set of Dune projects. A typical workspace consists of a
+single Dune project.
+
+A *Dune project* is defined by the presence of a
+:doc:`/reference/dune-project/index` file. Each Dune project extends over the
+file tree rooted at the directory containing the
+:doc:`/reference/dune-project/index` file, excluding any nested Dune projects.
+
+Dune determines the root of the current workspace by finding the topmost
+ancestor containing a :doc:`/reference/dune-project/index` file or by the
+presence of a :doc:`/reference/dune-workspace/index` file (see
+:ref:`finding-root` and :ref:`forcing-root` for details).
+
+Different Dune projects within the same Dune workspace are independent of each
+other and no settings are shared between them, even if they are nested within
+each other.
+
+Settings in :doc:`/reference/dune-workspace/index`, on the other hand, are
+inherited by all Dune projects in the workspace. Some settings (those that make
+sense for all projects) can be specified both in
+:doc:`/reference/dune-project/index` and :doc:`/reference/dune-workspace/index`
+files, with the former taking precedence. Note that all
+:doc:`/reference/dune-workspace/index` files other than the one specifying the
+root of the workspace are ignored.
+
+Within a Dune project, :doc:`/reference/dune/index` files are used to define all
+objects of interest for Dune: libraries, executables, tests, etc. There are
+typically many :doc:`/reference/dune/index` files in a Dune project: one per
+directory, unless the directory does not contain anything relevant to Dune. In
+each :doc:`/reference/dune/index` file, references are resolved relative to the
+directory containing the file.
+
+Note that there are specific stanzas and actions that may result in exceptions
+to some of the rules stated in the previous paragraph. See, for example, the
+:doc:`/reference/dune/subdir` and :doc:`/reference/dune/include_subdirs`
+stanzas, as well as the :doc:`/reference/actions/chdir` action.
+
+Finally, note that only public items (public libraries, public executables) of a
+Dune project are visible to other Dune projects within the same Dune workspace.

unikernel/duniverse/dune_/doc/explanation/tour/cli.rst

@@ -0,0 +1,29 @@
+Command-Line Interface
+----------------------
+
+The command-line interface is defined using `cmdliner
+<https://erratique.ch/software/cmdliner>`_. One thing to note is that we use
+binding operators to compose terms:
+
+`bin/print_rules.ml <https://github.com/ocaml/dune/blob/3.15.0/bin/print_rules.ml#L174-L190>`_
+  .. code-block:: ocaml
+    :linenos:
+    :lineno-start: 174
+
+    let+ builder = Common.Builder.term
+    and+ out =
+      Arg.(
+        value
+        & opt (some string) None
+        & info [ "o" ] ~docv:"FILE" ~doc:"Output to a file instead of stdout.")
+    and+ recursive =
+      Arg.(
+        value
+        & flag
+        & info
+            [ "r"; "recursive" ]
+            ~doc:
+              "Print all rules needed to build the transitive dependencies of the given \
+               targets.")
+    and+ syntax = Syntax.term
+    and+ targets = Arg.(value & pos_all dep [] & Arg.info [] ~docv:"TARGET") in

unikernel/duniverse/dune_/doc/explanation/tour/decoding.rst

@@ -0,0 +1,69 @@
+Parsing of Dune Files
+---------------------
+
+Parsing ``dune`` files is done in two steps:
+
+- They are parsed as S-expressions using :file:`src/dune_sexp/parser.mli`;
+- Then they are decoded using :file:`src/dune_sexp/decoder.mli`. The result of
+  this decoding step is added to an extensible variant using a mechanism in
+  :file:`src/dune_lang/stanza.mli`.
+
+Instead of writing a parser or using pattern matching, we define decoders,
+which are abstract values of type ``'a Decoder.t`` (returning a value of type
+``'a``). These decoders are assembled using combinators. For example, we can
+use simple decoders to write a decoder for a record type. This decoder
+abstraction is monadic, but the applicative subset is sufficient for most
+decoders.
+
+As an example, here is how ``(copy_files)`` is parsed:
+
+`src/dune_rules/stanzas/copy_files.ml <https://github.com/ocaml/dune/blob/3.15.0/src/dune_rules/stanzas/copy_files.ml#L31-L50>`_
+  .. code-block:: ocaml
+    :linenos:
+    :lineno-start: 31
+
+    let long_form =
+      let check = Dune_lang.Syntax.since Stanza.syntax (2, 7) in
+      let+ alias = field_o "alias" (check >>> Dune_lang.Alias.decode)
+      and+ mode = field "mode" ~default:Rule.Mode.Standard (check >>> Rule_mode_decoder.decode)
+      and+ enabled_if = Enabled_if.decode ~allowed_vars:Any ~since:(Some (2, 8)) ()
+      and+ files = field "files" (check >>> String_with_vars.decode)
+      and+ only_sources =
+        field_o
+          "only_sources"
+          (Dune_lang.Syntax.since Stanza.syntax (3, 14) >>> decode_only_sources)
+      and+ syntax_version = Dune_lang.Syntax.get_exn Stanza.syntax in
+      let only_sources = Option.value only_sources ~default:Blang.false_ in
+      { add_line_directive = false
+      ; alias
+      ; mode
+      ; enabled_if
+      ; files
+      ; only_sources
+      ; syntax_version
+      }
+
+The fields are queried individually, and a record is built using all the
+intermediate results. This will automatically take care of generating "unknown
+field X," "duplicate field X," and similar error messages.
+
+Another interesting thing to note is that the fields are not decoded directly,
+but use the following pattern:
+
+.. code:: ocaml
+
+   Syntax.since Stanza.syntax (x, y) >>> decoder
+
+Let's unpack this: ``(>>>)`` will run a ``unit Decoder.t`` on the input before
+passing the input to an actual decoder. The first decoder can be used to
+implement a check and trigger an error in some cases.
+
+Here, it is used for versioning. For example the ``(copy_files)`` stanza
+started supporting ``(enabled_if``) in version 2.8. Decoding this field is
+protected by this ``since`` call: it means that if the language version in
+:doc:`/reference/dune-project/index` file is greater than 2.8. In particular,
+this ensures that the project can not be built with Dune versions older than
+``2.8.0``.
+
+Once decoding succeeds, various stanzas are turned into various types defined
+in :file:`src/dune_rules/stanzas/`.

unikernel/duniverse/dune_/doc/explanation/tour/engine.rst

@@ -0,0 +1,15 @@
+The Engine
+==========
+
+The engine is the core, reusable part of Dune. It contains all the composable
+primitives that make it a build system.
+
+The fact that it is split from the :doc:`rules part <rule-generation>` makes it
+possible to create a different build system using this library. For example,
+Jane Street internally uses a build system with this engine as a backend, but a
+different frontend and CLI.
+
+In the context of Dune, the engine keeps track of the various directories and
+the rules in them and is able to build files using them. In addition, it takes
+care of the various caches that Dune uses, such as the one present in the
+``_build`` directory, the :doc:`shared cache </caching>`, etc.

unikernel/duniverse/dune_/doc/explanation/tour/index.rst

@@ -0,0 +1,34 @@
+A Tour of the Dune Codebase
+===========================
+
+.. note::
+
+  This document is based on Dune 3.15.0, whose source can be browsed `here
+  <https://github.com/ocaml/dune/tree/3.15.0>`_. The links in this tour point
+  to this version, but will not reflect how this works in other versions of
+  Dune.
+
+Let's start with a very high level tour of how ``dune build`` operates.
+
+As explained in :doc:`/explanation/mental-model`, ``dune build`` will interpret
+the targets listed on the command line, interpret the ``dune`` files in the
+workspace as rules, and execute the rules relevant to the requested targets.
+
+These steps correspond to areas of the Dune codebase:
+
+- the command-line interface is defined in :file:`bin/`;
+- the ``dune`` files are interpreted using a library defined in
+  :file:`src/dune_rules/`;
+- they are registered into an engine in :file:`src/dune_engine/`.
+
+Next, we will go deeper into these areas.
+
+.. toctree::
+
+  cli
+  decoding
+  rule-generation
+  engine
+  libraries
+  vendor
+  tests

unikernel/duniverse/dune_/doc/explanation/tour/libraries.rst

@@ -0,0 +1,12 @@
+Libraries
+=========
+
+Dune, as a package, is primarily an executable, but its source tree embeds a
+few public libraries. These are developed in :file:`otherlibs/`.
+
+Some of these have a special link to Dune, such as ``dune-build-info`` or
+``dune-site``. Others are just helper libraries that we develop as part of
+Dune, but they have a strong relation to the Dune internals, like ``dyn`` or
+``xdg``.
+
+.. seealso:: :doc:`/dune-libs`

unikernel/duniverse/dune_/doc/explanation/tour/rule-generation.rst

@@ -0,0 +1,168 @@
+Rule Generation
+---------------
+
+Using these parsed stanzas, the next step is to generate rules. This work
+starts in :file:`src/dune_rules/gen_rules.ml`, which dispatches to various
+modules in :file:`src/dune_rules/`.
+
+Rules are registered on the build engine using the following function from the
+``Super_context`` module:
+
+.. code-block:: ocaml
+
+  val add_rule
+    :  t
+    -> ?mode:Rule.Mode.t
+    -> ?loc:Loc.t
+    -> dir:Path.Build.t
+    -> Action.Full.t Action_builder.With_targets.t
+    -> unit Memo.t
+
+A value of ``Super_context.t`` represents an OCaml toolchain (``Context.t``) as
+well as various capabilities to expand variables and refer to :doc:`(env)
+stanzas </reference/dune/env>`. The last, unlabelled argument corresponds to
+the fully annotated action. We'll go through its type below.
+
+The modules in :file:`src/dune_rules` often expose a function ``gen_rules``
+taking a parsed stanza, a ``Super_context.t`` value, a directory name (and
+other arguments), and returning ``unit Memo.t``.
+
+.. note::
+
+  The ``Memo`` module is central to how Dune operates. It is a monadic
+  memoization framework that allows two things:
+
+  - Sharing and caching expensive internal computations, such as computing the
+    list of libraries Dune knows about, or computing the list of flags that
+    should be used to compile a given module.
+  - Incremental recomputation of this cached data. ``Memo`` tracks dependencies
+    between memoized values and will only recompute the necessary ones when an
+    input changes. This is a mini in-memory build system that works like a
+    spreadsheet. It is essential to the watch mode.
+
+An example of rule is the :doc:`/reference/dune/mdx` stanza, implemented in
+:file:`src/dune_rules/mdx.ml`. There are several steps in setting up rules for
+a ``(mdx)`` stanza:
+
+- How to run ``ocaml-mdx deps`` on the input file to produce a ``.mdx.deps``
+- Run ``ocaml-mdx dune-gen`` to produce a ``mdx_gen.ml-gen`` OCaml source file
+- Compile this executable
+- Run this executable to produce a ``.corrected`` file
+- Register a :doc:`/reference/actions/diff` action between the ``.corrected`` file
+  and the original file
+
+Let's walk through these rules.
+
+The first one is about producing a ``.mdx.deps`` file. It is a simple call to
+``Super_context.add_rule``.
+
+.. code-block:: ocaml
+  :linenos:
+  :lineno-start: 312
+
+  let* () = Super_context.add_rule sctx ~loc ~dir (Deps.rule ~dir ~mdx_prog files)
+
+``Deps.rule`` is defined in a helper function:
+
+.. code-block:: ocaml
+  :linenos:
+  :lineno-start: 77
+
+  let rule ~dir ~mdx_prog (files : Files.t) =
+    Command.run_dyn_prog
+      ~dir:(Path.build dir)
+      mdx_prog
+      ~stdout_to:files.deps
+      [ Command.Args.A "deps"; Lazy.force color_always; Dep (Path.build files.Files.src) ]
+
+This is a rule made by just running a command, here ``mdx_prog`` (a resolved
+path to ``ocaml-mdx``, meaning it can point to a binary in ``PATH`` or a built
+version in the current workspace). Its arguments are a domain-specific language
+defined in :file:`src/dune_rules/command.mli` where ``A`` refers to a plain
+string, and ``Dep`` refers to a string that should be interpreted as a dependency.
+Between that, and the ``~stdout_to`` parameter, it is enough for Dune to know
+about the rule's dependencies (what it will read) and its target (what it will
+produce).
+
+The second rule, which generates ``mdx_gen.ml-gen``, is similar. It is also done
+by calling ``Command.run_dyn_prog``.
+
+The third rule, to build the executable, calls ``Exe.build_and_link`` that is a
+helper function.
+
+Let's observe how the fourth rule (that calls the generated executable) is set
+up.
+
+.. code-block:: ocaml
+
+    let mdx_action ~loc:_ =
+      let open Action_builder.With_targets.O in
+      let mdx_input_dependencies = (* ... *) in
+      let executable, command_line = (* ... *) in
+      let deps, sandbox = (* ... *) in
+      let+ action =
+        Action_builder.with_no_targets deps
+        >>> Action_builder.with_no_targets
+              (Action_builder.env_var "MDX_RUN_NON_DETERMINISTIC")
+        >>> Action_builder.with_no_targets
+              (Action_builder.map mdx_input_dependencies ~f:(fun d -> (), d)
+               |> Action_builder.dyn_deps)
+        >>> Command.run_dyn_prog
+              ~dir:(Path.build dir)
+              ~stdout_to:files.corrected
+              executable
+              command_line
+      and+ locks =
+        Expander.expand_locks expander stanza.locks |> Action_builder.with_no_targets
+      in
+      Action.Full.add_locks locks action |> Action.Full.add_sandbox sandbox
+    in
+    Super_context.add_rule sctx ~loc ~dir (mdx_action ~loc)
+
+Here, the ``mdx_action`` that is set up is not just a single
+``Command.run_dyn_prog`` call. It is assembled using combinators from
+``Action_builder.With_targets``. This is another monad used in Dune. It
+corresponds to what can happen at build time, like running commands or creating
+files, or more complex actions such as reading a file that needs to be built by
+another rule. It is also used to track dependencies and targets. The "thing"
+that we register to the Dune engine using ``Super_context.add_rule`` has type
+``Action.Full.t Action_builder.With_targets.t``.
+
+.. note::
+
+  This is different from ``Memo``, which corresponds to what happens within
+  Dune itself. But it is also possible to use ``Memo`` from an
+  ``Action_builder`` context. In that sense, ``Action_builder`` is more
+  powerful: at execution time, ``Action_builder`` will manage what happens in
+  the ``_build`` directory, while ``Memo`` is only concerned with what happens
+  in memory.
+
+Finally, to register the correction, the technique is to attach the
+:doc:`/reference/actions/diff` action to the :doc:`/reference/aliases/runtest`
+alias (a collection of rules) using this call:
+
+.. code-block:: ocaml
+  :linenos:
+  :lineno-start: 405
+
+  (* Attach the diff action to the @runtest for the src and corrected files *)
+  Files.diff_action files
+  |> Super_context.add_alias_action sctx (Alias.make Alias0.runtest ~dir) ~loc ~dir
+
+Where ``Files.diff_action`` is defined as:
+
+.. code-block:: ocaml
+  :linenos:
+  :lineno-start: 33
+
+  let diff_action { src; corrected; deps = _ } =
+    let src = Path.build src in
+    let open Action_builder.O in
+    let+ () = Action_builder.path src
+    and+ () = Action_builder.path (Path.build corrected) in
+    Action.Full.make (Action.diff ~optional:false src corrected)
+  ;;
+
+As explained above, ``Action_builder`` keeps tracks of dependencies, so using
+``let+ () = Action_builder.path src`` is a way to declare ``src`` as a
+dependency of the current action.

unikernel/duniverse/dune_/doc/explanation/tour/tests.rst

@@ -0,0 +1,33 @@
+Tests
+=====
+
+The :file:`test/` directory contains all the tests for Dune itself. Additionally,
+the tests for our :doc:`libraries` are stored in :file:`otherlibs/` next to the
+library itself.
+
+We have 3 kind of tests:
+
+- Unit tests, in :file:`test/unit-tests` (we have very few of these, usually
+  preferring other kinds)
+- Expect tests, in :file:`test/expect-tests` (using ``ppx_expect``)
+- :doc:`Cram tests </reference/cram>`, in :file:`test/blackbox-tests/`. This is
+  our preferred way of testing.
+
+The actual Cram tests are in :file:`test/expect-tests/test-cases`. There is a
+mix of file tests and directory tests. For regression tests, the pattern
+``githubNUMBER.t`` is used.
+
+The ``dune`` file at :file:`test/expect-tests/test-cases/dune` sets up some
+metadata for the tests. For example, if a test has an external dependency like
+``strace``, a dependency on ``%{bin:strace}`` will prevent the test from even
+trying to start. Some tests are also disabled on some configurations using
+``(enabled_if)``.
+
+Finally, some programs available in the Cram tests are defined in
+:file:`test/expect-tests/blackbox-tests/utils`. For example, we have `a
+dune_cmd program
+<https://github.com/ocaml/dune/blob/3.15.0/test/blackbox-tests/utils/dune_cmd.ml>`_
+that contains reimplementations of common utilities like ``stat``, which do not
+have the same output on the different systems we use to test Dune.
+
+.. seealso:: :doc:`/hacking`

unikernel/duniverse/dune_/doc/explanation/tour/vendor.rst

@@ -0,0 +1,15 @@
+Vendored Libraries
+==================
+
+As an opam package, Dune has no dependencies. But it uses some existing
+libraries by copying, or "vendoring", their source code into the
+:file:`vendor/` directory.
+
+In some cases, the external dependency is extracted from the upstream
+repository. In other cases, we carry patches and refer to a fork in the
+`ocaml-dune GitHub organization <https://github.com/ocaml-dune>`_.
+
+The source code in the :file:`vendor/` directory is not meant to be edited
+directly. Instead, it is edited in the external repository, and the copy in the
+Dune source tree is updated by running an update script, such as
+`update-spawn.sh <https://github.com/ocaml/dune/blob/3.15.0/vendor/update-spawn.sh>`_.