unikernel/duniverse/dune_/doc/explanation/tour/cli.rst

@@ -0,0 +1,29 @@
+Command-Line Interface
+----------------------
+
+The command-line interface is defined using `cmdliner
+<https://erratique.ch/software/cmdliner>`_. One thing to note is that we use
+binding operators to compose terms:
+
+`bin/print_rules.ml <https://github.com/ocaml/dune/blob/3.15.0/bin/print_rules.ml#L174-L190>`_
+  .. code-block:: ocaml
+    :linenos:
+    :lineno-start: 174
+
+    let+ builder = Common.Builder.term
+    and+ out =
+      Arg.(
+        value
+        & opt (some string) None
+        & info [ "o" ] ~docv:"FILE" ~doc:"Output to a file instead of stdout.")
+    and+ recursive =
+      Arg.(
+        value
+        & flag
+        & info
+            [ "r"; "recursive" ]
+            ~doc:
+              "Print all rules needed to build the transitive dependencies of the given \
+               targets.")
+    and+ syntax = Syntax.term
+    and+ targets = Arg.(value & pos_all dep [] & Arg.info [] ~docv:"TARGET") in

unikernel/duniverse/dune_/doc/explanation/tour/decoding.rst

@@ -0,0 +1,69 @@
+Parsing of Dune Files
+---------------------
+
+Parsing ``dune`` files is done in two steps:
+
+- They are parsed as S-expressions using :file:`src/dune_sexp/parser.mli`;
+- Then they are decoded using :file:`src/dune_sexp/decoder.mli`. The result of
+  this decoding step is added to an extensible variant using a mechanism in
+  :file:`src/dune_lang/stanza.mli`.
+
+Instead of writing a parser or using pattern matching, we define decoders,
+which are abstract values of type ``'a Decoder.t`` (returning a value of type
+``'a``). These decoders are assembled using combinators. For example, we can
+use simple decoders to write a decoder for a record type. This decoder
+abstraction is monadic, but the applicative subset is sufficient for most
+decoders.
+
+As an example, here is how ``(copy_files)`` is parsed:
+
+`src/dune_rules/stanzas/copy_files.ml <https://github.com/ocaml/dune/blob/3.15.0/src/dune_rules/stanzas/copy_files.ml#L31-L50>`_
+  .. code-block:: ocaml
+    :linenos:
+    :lineno-start: 31
+
+    let long_form =
+      let check = Dune_lang.Syntax.since Stanza.syntax (2, 7) in
+      let+ alias = field_o "alias" (check >>> Dune_lang.Alias.decode)
+      and+ mode = field "mode" ~default:Rule.Mode.Standard (check >>> Rule_mode_decoder.decode)
+      and+ enabled_if = Enabled_if.decode ~allowed_vars:Any ~since:(Some (2, 8)) ()
+      and+ files = field "files" (check >>> String_with_vars.decode)
+      and+ only_sources =
+        field_o
+          "only_sources"
+          (Dune_lang.Syntax.since Stanza.syntax (3, 14) >>> decode_only_sources)
+      and+ syntax_version = Dune_lang.Syntax.get_exn Stanza.syntax in
+      let only_sources = Option.value only_sources ~default:Blang.false_ in
+      { add_line_directive = false
+      ; alias
+      ; mode
+      ; enabled_if
+      ; files
+      ; only_sources
+      ; syntax_version
+      }
+
+The fields are queried individually, and a record is built using all the
+intermediate results. This will automatically take care of generating "unknown
+field X," "duplicate field X," and similar error messages.
+
+Another interesting thing to note is that the fields are not decoded directly,
+but use the following pattern:
+
+.. code:: ocaml
+
+   Syntax.since Stanza.syntax (x, y) >>> decoder
+
+Let's unpack this: ``(>>>)`` will run a ``unit Decoder.t`` on the input before
+passing the input to an actual decoder. The first decoder can be used to
+implement a check and trigger an error in some cases.
+
+Here, it is used for versioning. For example the ``(copy_files)`` stanza
+started supporting ``(enabled_if``) in version 2.8. Decoding this field is
+protected by this ``since`` call: it means that if the language version in
+:doc:`/reference/dune-project/index` file is greater than 2.8. In particular,
+this ensures that the project can not be built with Dune versions older than
+``2.8.0``.
+
+Once decoding succeeds, various stanzas are turned into various types defined
+in :file:`src/dune_rules/stanzas/`.

unikernel/duniverse/dune_/doc/explanation/tour/engine.rst

@@ -0,0 +1,15 @@
+The Engine
+==========
+
+The engine is the core, reusable part of Dune. It contains all the composable
+primitives that make it a build system.
+
+The fact that it is split from the :doc:`rules part <rule-generation>` makes it
+possible to create a different build system using this library. For example,
+Jane Street internally uses a build system with this engine as a backend, but a
+different frontend and CLI.
+
+In the context of Dune, the engine keeps track of the various directories and
+the rules in them and is able to build files using them. In addition, it takes
+care of the various caches that Dune uses, such as the one present in the
+``_build`` directory, the :doc:`shared cache </caching>`, etc.

unikernel/duniverse/dune_/doc/explanation/tour/index.rst

@@ -0,0 +1,34 @@
+A Tour of the Dune Codebase
+===========================
+
+.. note::
+
+  This document is based on Dune 3.15.0, whose source can be browsed `here
+  <https://github.com/ocaml/dune/tree/3.15.0>`_. The links in this tour point
+  to this version, but will not reflect how this works in other versions of
+  Dune.
+
+Let's start with a very high level tour of how ``dune build`` operates.
+
+As explained in :doc:`/explanation/mental-model`, ``dune build`` will interpret
+the targets listed on the command line, interpret the ``dune`` files in the
+workspace as rules, and execute the rules relevant to the requested targets.
+
+These steps correspond to areas of the Dune codebase:
+
+- the command-line interface is defined in :file:`bin/`;
+- the ``dune`` files are interpreted using a library defined in
+  :file:`src/dune_rules/`;
+- they are registered into an engine in :file:`src/dune_engine/`.
+
+Next, we will go deeper into these areas.
+
+.. toctree::
+
+  cli
+  decoding
+  rule-generation
+  engine
+  libraries
+  vendor
+  tests

unikernel/duniverse/dune_/doc/explanation/tour/libraries.rst

@@ -0,0 +1,12 @@
+Libraries
+=========
+
+Dune, as a package, is primarily an executable, but its source tree embeds a
+few public libraries. These are developed in :file:`otherlibs/`.
+
+Some of these have a special link to Dune, such as ``dune-build-info`` or
+``dune-site``. Others are just helper libraries that we develop as part of
+Dune, but they have a strong relation to the Dune internals, like ``dyn`` or
+``xdg``.
+
+.. seealso:: :doc:`/dune-libs`

unikernel/duniverse/dune_/doc/explanation/tour/rule-generation.rst

@@ -0,0 +1,168 @@
+Rule Generation
+---------------
+
+Using these parsed stanzas, the next step is to generate rules. This work
+starts in :file:`src/dune_rules/gen_rules.ml`, which dispatches to various
+modules in :file:`src/dune_rules/`.
+
+Rules are registered on the build engine using the following function from the
+``Super_context`` module:
+
+.. code-block:: ocaml
+
+  val add_rule
+    :  t
+    -> ?mode:Rule.Mode.t
+    -> ?loc:Loc.t
+    -> dir:Path.Build.t
+    -> Action.Full.t Action_builder.With_targets.t
+    -> unit Memo.t
+
+A value of ``Super_context.t`` represents an OCaml toolchain (``Context.t``) as
+well as various capabilities to expand variables and refer to :doc:`(env)
+stanzas </reference/dune/env>`. The last, unlabelled argument corresponds to
+the fully annotated action. We'll go through its type below.
+
+The modules in :file:`src/dune_rules` often expose a function ``gen_rules``
+taking a parsed stanza, a ``Super_context.t`` value, a directory name (and
+other arguments), and returning ``unit Memo.t``.
+
+.. note::
+
+  The ``Memo`` module is central to how Dune operates. It is a monadic
+  memoization framework that allows two things:
+
+  - Sharing and caching expensive internal computations, such as computing the
+    list of libraries Dune knows about, or computing the list of flags that
+    should be used to compile a given module.
+  - Incremental recomputation of this cached data. ``Memo`` tracks dependencies
+    between memoized values and will only recompute the necessary ones when an
+    input changes. This is a mini in-memory build system that works like a
+    spreadsheet. It is essential to the watch mode.
+
+An example of rule is the :doc:`/reference/dune/mdx` stanza, implemented in
+:file:`src/dune_rules/mdx.ml`. There are several steps in setting up rules for
+a ``(mdx)`` stanza:
+
+- How to run ``ocaml-mdx deps`` on the input file to produce a ``.mdx.deps``
+- Run ``ocaml-mdx dune-gen`` to produce a ``mdx_gen.ml-gen`` OCaml source file
+- Compile this executable
+- Run this executable to produce a ``.corrected`` file
+- Register a :doc:`/reference/actions/diff` action between the ``.corrected`` file
+  and the original file
+
+Let's walk through these rules.
+
+The first one is about producing a ``.mdx.deps`` file. It is a simple call to
+``Super_context.add_rule``.
+
+.. code-block:: ocaml
+  :linenos:
+  :lineno-start: 312
+
+  let* () = Super_context.add_rule sctx ~loc ~dir (Deps.rule ~dir ~mdx_prog files)
+
+``Deps.rule`` is defined in a helper function:
+
+.. code-block:: ocaml
+  :linenos:
+  :lineno-start: 77
+
+  let rule ~dir ~mdx_prog (files : Files.t) =
+    Command.run_dyn_prog
+      ~dir:(Path.build dir)
+      mdx_prog
+      ~stdout_to:files.deps
+      [ Command.Args.A "deps"; Lazy.force color_always; Dep (Path.build files.Files.src) ]
+
+This is a rule made by just running a command, here ``mdx_prog`` (a resolved
+path to ``ocaml-mdx``, meaning it can point to a binary in ``PATH`` or a built
+version in the current workspace). Its arguments are a domain-specific language
+defined in :file:`src/dune_rules/command.mli` where ``A`` refers to a plain
+string, and ``Dep`` refers to a string that should be interpreted as a dependency.
+Between that, and the ``~stdout_to`` parameter, it is enough for Dune to know
+about the rule's dependencies (what it will read) and its target (what it will
+produce).
+
+The second rule, which generates ``mdx_gen.ml-gen``, is similar. It is also done
+by calling ``Command.run_dyn_prog``.
+
+The third rule, to build the executable, calls ``Exe.build_and_link`` that is a
+helper function.
+
+Let's observe how the fourth rule (that calls the generated executable) is set
+up.
+
+.. code-block:: ocaml
+
+    let mdx_action ~loc:_ =
+      let open Action_builder.With_targets.O in
+      let mdx_input_dependencies = (* ... *) in
+      let executable, command_line = (* ... *) in
+      let deps, sandbox = (* ... *) in
+      let+ action =
+        Action_builder.with_no_targets deps
+        >>> Action_builder.with_no_targets
+              (Action_builder.env_var "MDX_RUN_NON_DETERMINISTIC")
+        >>> Action_builder.with_no_targets
+              (Action_builder.map mdx_input_dependencies ~f:(fun d -> (), d)
+               |> Action_builder.dyn_deps)
+        >>> Command.run_dyn_prog
+              ~dir:(Path.build dir)
+              ~stdout_to:files.corrected
+              executable
+              command_line
+      and+ locks =
+        Expander.expand_locks expander stanza.locks |> Action_builder.with_no_targets
+      in
+      Action.Full.add_locks locks action |> Action.Full.add_sandbox sandbox
+    in
+    Super_context.add_rule sctx ~loc ~dir (mdx_action ~loc)
+
+Here, the ``mdx_action`` that is set up is not just a single
+``Command.run_dyn_prog`` call. It is assembled using combinators from
+``Action_builder.With_targets``. This is another monad used in Dune. It
+corresponds to what can happen at build time, like running commands or creating
+files, or more complex actions such as reading a file that needs to be built by
+another rule. It is also used to track dependencies and targets. The "thing"
+that we register to the Dune engine using ``Super_context.add_rule`` has type
+``Action.Full.t Action_builder.With_targets.t``.
+
+.. note::
+
+  This is different from ``Memo``, which corresponds to what happens within
+  Dune itself. But it is also possible to use ``Memo`` from an
+  ``Action_builder`` context. In that sense, ``Action_builder`` is more
+  powerful: at execution time, ``Action_builder`` will manage what happens in
+  the ``_build`` directory, while ``Memo`` is only concerned with what happens
+  in memory.
+
+Finally, to register the correction, the technique is to attach the
+:doc:`/reference/actions/diff` action to the :doc:`/reference/aliases/runtest`
+alias (a collection of rules) using this call:
+
+.. code-block:: ocaml
+  :linenos:
+  :lineno-start: 405
+
+  (* Attach the diff action to the @runtest for the src and corrected files *)
+  Files.diff_action files
+  |> Super_context.add_alias_action sctx (Alias.make Alias0.runtest ~dir) ~loc ~dir
+
+Where ``Files.diff_action`` is defined as:
+
+.. code-block:: ocaml
+  :linenos:
+  :lineno-start: 33
+
+  let diff_action { src; corrected; deps = _ } =
+    let src = Path.build src in
+    let open Action_builder.O in
+    let+ () = Action_builder.path src
+    and+ () = Action_builder.path (Path.build corrected) in
+    Action.Full.make (Action.diff ~optional:false src corrected)
+  ;;
+
+As explained above, ``Action_builder`` keeps tracks of dependencies, so using
+``let+ () = Action_builder.path src`` is a way to declare ``src`` as a
+dependency of the current action.

unikernel/duniverse/dune_/doc/explanation/tour/tests.rst

@@ -0,0 +1,33 @@
+Tests
+=====
+
+The :file:`test/` directory contains all the tests for Dune itself. Additionally,
+the tests for our :doc:`libraries` are stored in :file:`otherlibs/` next to the
+library itself.
+
+We have 3 kind of tests:
+
+- Unit tests, in :file:`test/unit-tests` (we have very few of these, usually
+  preferring other kinds)
+- Expect tests, in :file:`test/expect-tests` (using ``ppx_expect``)
+- :doc:`Cram tests </reference/cram>`, in :file:`test/blackbox-tests/`. This is
+  our preferred way of testing.
+
+The actual Cram tests are in :file:`test/expect-tests/test-cases`. There is a
+mix of file tests and directory tests. For regression tests, the pattern
+``githubNUMBER.t`` is used.
+
+The ``dune`` file at :file:`test/expect-tests/test-cases/dune` sets up some
+metadata for the tests. For example, if a test has an external dependency like
+``strace``, a dependency on ``%{bin:strace}`` will prevent the test from even
+trying to start. Some tests are also disabled on some configurations using
+``(enabled_if)``.
+
+Finally, some programs available in the Cram tests are defined in
+:file:`test/expect-tests/blackbox-tests/utils`. For example, we have `a
+dune_cmd program
+<https://github.com/ocaml/dune/blob/3.15.0/test/blackbox-tests/utils/dune_cmd.ml>`_
+that contains reimplementations of common utilities like ``stat``, which do not
+have the same output on the different systems we use to test Dune.
+
+.. seealso:: :doc:`/hacking`

unikernel/duniverse/dune_/doc/explanation/tour/vendor.rst

@@ -0,0 +1,15 @@
+Vendored Libraries
+==================
+
+As an opam package, Dune has no dependencies. But it uses some existing
+libraries by copying, or "vendoring", their source code into the
+:file:`vendor/` directory.
+
+In some cases, the external dependency is extracted from the upstream
+repository. In other cases, we carry patches and refer to a fork in the
+`ocaml-dune GitHub organization <https://github.com/ocaml-dune>`_.
+
+The source code in the :file:`vendor/` directory is not meant to be edited
+directly. Instead, it is edited in the external repository, and the copy in the
+Dune source tree is updated by running an update script, such as
+`update-spawn.sh <https://github.com/ocaml/dune/blob/3.15.0/vendor/update-spawn.sh>`_.