unikernel/duniverse/dune_/doc/papers/ocaml-2022/memo.md

@@ -0,0 +1,151 @@
+---
+geometry: "left=2cm,right=2cm,top=2.4cm,bottom=2.1cm"
+bibliography: refs.bib
+csl: refs.csl
+---
+
+# Memo: an incremental computation library that powers Dune
+
+Andrey Mokhov, Arseniy Alekseyev
+
+*Jane Street, London, United Kingdom*
+
+### Abstract
+
+We present Memo, an incremental computation library that supports a new, faster
+and more scalable, file-watching build mode in Dune 3.0. The requirements from
+the build systems domain make Memo a unique point in the design space of
+incremental computation libraries. Specifically, Memo needs to cope with
+concurrency, dynamic dependencies, dependency cycles, and non-determinism;
+provide support for efficiently collecting and reporting user-friendly errors;
+and scale to computation graphs containing tens of millions of incremental
+nodes.
+
+## Introduction
+
+The OCaml build system Dune [@dune] supports a continuous file-watching build
+mode, where rebuilds are triggered automatically as the user is editing source
+files. A simple and naive way to implement this functionality is to restart Dune
+on any file change but that is too slow for large projects. To avoid re-scanning
+the project's source tree, re-parsing all build specification files, and
+re-generating all build rules from scratch every time, Dune uses an incremental
+computation library called Memo. Below we briefly introduce the Memo's API.
+
+Memo provides a monadic API built on top of the *structured concurrency monad*
+`Fiber`. Dune uses `Fiber` to speed up builds by executing external commands in
+parallel, e.g., running multiple instances of `ocamlopt` to compile independent
+source files. To inject a *fiber* into the `Memo` monad, and to extract it back,
+one can use the following pair of functions:
+
+```ocaml
+  val of_fiber : 'a Fiber.t -> 'a Memo.t
+  val run : 'a Memo.t -> 'a Fiber.t
+```
+
+More interestingly, functions in the `Memo` monad can be memoized and cached
+between different build runs[^1]:
+
+[^1]: The function `create` takes a few more arguments, e.g., for reporting good
+error messages, which we omit for the sake of clarity.
+
+```ocaml
+  val create : ('i -> 'o Memo.t) -> ('i, 'o) Memo.Table.t
+  val exec : ('i, 'o) Memo.Table.t -> 'i -> 'o Memo.t
+```
+
+Here `Memo.Table.t` is a table that stores the input/output mapping computed in
+the current run, along with the *dependencies* that are automatically captured
+when memoized functions call one another. Such explicit memoization, instead of,
+e.g., memoizing every `Memo.map` and `Memo.bind` call, makes it easy to control
+the degree of incrementality.
+
+Finally, Memo provides a way to *invalidate* a specific input/output pair, or a
+*cell*, via the following API:
+
+```ocaml
+  val cell : ('i, 'o) Memo.Table.t -> 'i -> ('i, 'o) Cell.t
+  val invalidate : ('i, 'o) Cell.t -> Invalidation.t (* [Invalidation.t]s can be combined *)
+  val restart : Invalidation.t -> unit
+```
+
+When Dune receives new events from the file-watching backend, it cancels the
+build run by interrupting the currently running external commands (if any), and
+then calls `restart` to let Memo know which files changed and initiate a new
+build run. When evaluating future calls to `run`, Memo will consider all outputs
+that transitively depend on the invalidated cells as out of date, and will
+recompute them when/if needed.
+
+## Key features
+
+This section discusses the most interesting features of Memo and some aspects of
+the implementation.
+
+Firstly, to *capture dependencies*, Memo maintains a call stack, where stack
+frames correspond to `exec` calls. The call stack is also used for reporting
+good error messages: when a user-supplied function raises an exception, we
+extend it with the current Memo stack trace using human-readable annotations
+provided to `create` via an optional argument.
+
+Memo supports two ways of *error reporting*: *early* (to show errors to the user
+as soon as they occur during a build), and *deterministic* (to provide a stable
+error summary at the end of the build). To speed up rebuilds, Memo also supports
+*error caching*: by default, it doesn't recompute a `Memo` function that
+previously failed if its dependencies are up to date. This behaviour can be
+overridden for *non-reproducible errors* that should not be cached, for example,
+the errors that occur while Dune cancels the current build by interrupting the
+execution of external commands.
+
+One of the most interesting features of Memo compared to other incremental
+computation libraries is *dependency cycle detection*.
+Since Dune supports *dynamic build dependencies* [@mokhov2020build], the
+dependency graph is not known before the build starts. During the build, new
+computation nodes and dependency edges are discovered concurrently, and Memo
+uses an incremental cycle detection algorithm [@gueneau2019cycles] to detect and
+report dependency cycles as soon as they are created. This is a unique feature
+of Memo: Incremental [@incremental] and Adapton [@hammer2014adapton] libraries
+do not support concurrency; the Tenacious library (used by Jane Street's
+internal build system Jenga) does support concurrency but it detects cycles by
+"stopping the world" and traversing the "frozen" dependency graph to see if
+concurrent computations might have deadlocked by waiting for each other. The
+approach used by Tenacious is conceptually simple but introduces a delay between
+the creation of a dependency cycle and its detection. Memo reports dependency
+cycle errors without delay, and uses the human-readable annotations supplied to
+`create` to make it easier to understand and debug cycles.
+
+Compared to the incremental computation libraries mentioned above, the current
+implementation of Memo makes one unusual design choice. Incremental, Adapton and
+Tenacious are all *push based*, i.e., they trigger recomputation starting from
+the leaves of the computation graph. Memo is *pull based*: `invalidate` calls
+merely mark leaves (cells) as out of date, but recomputation is driven from the
+top-level `run` calls. The main drawback of our approach is that Memo traverses
+the whole graph on each rebuild. The main benefits are: (i) Memo
+doesn't need to store reverse dependencies, which saves space and eliminates
+various garbage collection pitfalls; (ii) the pull based approach makes it
+easier to avoid "spurious" recomputations, where a node is recomputed but
+subsequently becomes unreachable from the top due to a new dependency structure.
+Our current experiments show that traversing the whole graph on each rebuild
+isn't prohibitively expensive even for large builds where computation graphs
+contain tens of millions of nodes. We may rethink this design decision in future
+as Dune and Memo need to scale to larger projects. Having said that, we consider
+the pull based approach to incremental computation to be under-researched, and
+are keen to investigate how far we can take it in practice.
+
+## Development status
+
+Memo is still in active development and we welcome feedback from the OCaml
+community on how to make it better. While the current implementation is tied to
+Dune's lightweight concurrency library Fiber, the core functionality can be made
+available as a functor over an arbitrary concurrency monad, making it usable
+with Async and Lwt.
+
+## Acknowledgements
+
+We thank Jeremie Dimino for driving the design of Memo and for his work on
+incrementalising Dune. We are also grateful to Rudi Horn, Rudi Grinberg, Emilio Jesús Gallego
+Arias and other Dune developers for their many contributions, and to Armaël
+Guéneau for helping us integrate the incremental cycle detection library in
+Memo.
+
+# References
+
+<!-- References to be generated by Pandoc. -->