unikernel/duniverse/dune_/doc/papers/ocaml-2022/README.md

@@ -0,0 +1,4 @@
+A paper describing the Memo library presented at the OCaml Users and
+Developers Workshop 2022.
+
+To render a PDF version, run `pandoc memo.md --citeproc -H header.tex -o memo.pdf`.

unikernel/duniverse/dune_/doc/papers/ocaml-2022/header.tex

@@ -0,0 +1,10 @@
+% Run Pandoc with -H header.tex to apply these changes
+
+\usepackage{url}
+
+\definecolor{inline_code_colour}{HTML}{000080}
+
+\let\oldtexttt\texttt
+\renewcommand{\texttt}[1]{\textcolor{inline_code_colour}{\oldtexttt{#1}}}
+
+\urlstyle{sf}

unikernel/duniverse/dune_/doc/papers/ocaml-2022/memo.md

@@ -0,0 +1,151 @@
+---
+geometry: "left=2cm,right=2cm,top=2.4cm,bottom=2.1cm"
+bibliography: refs.bib
+csl: refs.csl
+---
+
+# Memo: an incremental computation library that powers Dune
+
+Andrey Mokhov, Arseniy Alekseyev
+
+*Jane Street, London, United Kingdom*
+
+### Abstract
+
+We present Memo, an incremental computation library that supports a new, faster
+and more scalable, file-watching build mode in Dune 3.0. The requirements from
+the build systems domain make Memo a unique point in the design space of
+incremental computation libraries. Specifically, Memo needs to cope with
+concurrency, dynamic dependencies, dependency cycles, and non-determinism;
+provide support for efficiently collecting and reporting user-friendly errors;
+and scale to computation graphs containing tens of millions of incremental
+nodes.
+
+## Introduction
+
+The OCaml build system Dune [@dune] supports a continuous file-watching build
+mode, where rebuilds are triggered automatically as the user is editing source
+files. A simple and naive way to implement this functionality is to restart Dune
+on any file change but that is too slow for large projects. To avoid re-scanning
+the project's source tree, re-parsing all build specification files, and
+re-generating all build rules from scratch every time, Dune uses an incremental
+computation library called Memo. Below we briefly introduce the Memo's API.
+
+Memo provides a monadic API built on top of the *structured concurrency monad*
+`Fiber`. Dune uses `Fiber` to speed up builds by executing external commands in
+parallel, e.g., running multiple instances of `ocamlopt` to compile independent
+source files. To inject a *fiber* into the `Memo` monad, and to extract it back,
+one can use the following pair of functions:
+
+```ocaml
+  val of_fiber : 'a Fiber.t -> 'a Memo.t
+  val run : 'a Memo.t -> 'a Fiber.t
+```
+
+More interestingly, functions in the `Memo` monad can be memoized and cached
+between different build runs[^1]:
+
+[^1]: The function `create` takes a few more arguments, e.g., for reporting good
+error messages, which we omit for the sake of clarity.
+
+```ocaml
+  val create : ('i -> 'o Memo.t) -> ('i, 'o) Memo.Table.t
+  val exec : ('i, 'o) Memo.Table.t -> 'i -> 'o Memo.t
+```
+
+Here `Memo.Table.t` is a table that stores the input/output mapping computed in
+the current run, along with the *dependencies* that are automatically captured
+when memoized functions call one another. Such explicit memoization, instead of,
+e.g., memoizing every `Memo.map` and `Memo.bind` call, makes it easy to control
+the degree of incrementality.
+
+Finally, Memo provides a way to *invalidate* a specific input/output pair, or a
+*cell*, via the following API:
+
+```ocaml
+  val cell : ('i, 'o) Memo.Table.t -> 'i -> ('i, 'o) Cell.t
+  val invalidate : ('i, 'o) Cell.t -> Invalidation.t (* [Invalidation.t]s can be combined *)
+  val restart : Invalidation.t -> unit
+```
+
+When Dune receives new events from the file-watching backend, it cancels the
+build run by interrupting the currently running external commands (if any), and
+then calls `restart` to let Memo know which files changed and initiate a new
+build run. When evaluating future calls to `run`, Memo will consider all outputs
+that transitively depend on the invalidated cells as out of date, and will
+recompute them when/if needed.
+
+## Key features
+
+This section discusses the most interesting features of Memo and some aspects of
+the implementation.
+
+Firstly, to *capture dependencies*, Memo maintains a call stack, where stack
+frames correspond to `exec` calls. The call stack is also used for reporting
+good error messages: when a user-supplied function raises an exception, we
+extend it with the current Memo stack trace using human-readable annotations
+provided to `create` via an optional argument.
+
+Memo supports two ways of *error reporting*: *early* (to show errors to the user
+as soon as they occur during a build), and *deterministic* (to provide a stable
+error summary at the end of the build). To speed up rebuilds, Memo also supports
+*error caching*: by default, it doesn't recompute a `Memo` function that
+previously failed if its dependencies are up to date. This behaviour can be
+overridden for *non-reproducible errors* that should not be cached, for example,
+the errors that occur while Dune cancels the current build by interrupting the
+execution of external commands.
+
+One of the most interesting features of Memo compared to other incremental
+computation libraries is *dependency cycle detection*.
+Since Dune supports *dynamic build dependencies* [@mokhov2020build], the
+dependency graph is not known before the build starts. During the build, new
+computation nodes and dependency edges are discovered concurrently, and Memo
+uses an incremental cycle detection algorithm [@gueneau2019cycles] to detect and
+report dependency cycles as soon as they are created. This is a unique feature
+of Memo: Incremental [@incremental] and Adapton [@hammer2014adapton] libraries
+do not support concurrency; the Tenacious library (used by Jane Street's
+internal build system Jenga) does support concurrency but it detects cycles by
+"stopping the world" and traversing the "frozen" dependency graph to see if
+concurrent computations might have deadlocked by waiting for each other. The
+approach used by Tenacious is conceptually simple but introduces a delay between
+the creation of a dependency cycle and its detection. Memo reports dependency
+cycle errors without delay, and uses the human-readable annotations supplied to
+`create` to make it easier to understand and debug cycles.
+
+Compared to the incremental computation libraries mentioned above, the current
+implementation of Memo makes one unusual design choice. Incremental, Adapton and
+Tenacious are all *push based*, i.e., they trigger recomputation starting from
+the leaves of the computation graph. Memo is *pull based*: `invalidate` calls
+merely mark leaves (cells) as out of date, but recomputation is driven from the
+top-level `run` calls. The main drawback of our approach is that Memo traverses
+the whole graph on each rebuild. The main benefits are: (i) Memo
+doesn't need to store reverse dependencies, which saves space and eliminates
+various garbage collection pitfalls; (ii) the pull based approach makes it
+easier to avoid "spurious" recomputations, where a node is recomputed but
+subsequently becomes unreachable from the top due to a new dependency structure.
+Our current experiments show that traversing the whole graph on each rebuild
+isn't prohibitively expensive even for large builds where computation graphs
+contain tens of millions of nodes. We may rethink this design decision in future
+as Dune and Memo need to scale to larger projects. Having said that, we consider
+the pull based approach to incremental computation to be under-researched, and
+are keen to investigate how far we can take it in practice.
+
+## Development status
+
+Memo is still in active development and we welcome feedback from the OCaml
+community on how to make it better. While the current implementation is tied to
+Dune's lightweight concurrency library Fiber, the core functionality can be made
+available as a functor over an arbitrary concurrency monad, making it usable
+with Async and Lwt.
+
+## Acknowledgements
+
+We thank Jeremie Dimino for driving the design of Memo and for his work on
+incrementalising Dune. We are also grateful to Rudi Horn, Rudi Grinberg, Emilio Jesús Gallego
+Arias and other Dune developers for their many contributions, and to Armaël
+Guéneau for helping us integrate the incremental cycle detection library in
+Memo.
+
+# References
+
+<!-- References to be generated by Pandoc. -->

unikernel/duniverse/dune_/doc/papers/ocaml-2022/refs.bib

@@ -0,0 +1,43 @@
+@unpublished{dune,
+    title = {{Dune: A composable build system}},
+    author = {{Jane~Street}},
+    year = {2018},
+    note = "\url{https://dune.build}"
+}
+
+@article{mokhov2020build,
+  title={Build systems {\`a} la carte: Theory and practice},
+  author={Mokhov, Andrey and Mitchell, Neil and Peyton Jones, Simon},
+  journal={Journal of Functional Programming},
+  volume={30},
+  year={2020},
+  publisher={Cambridge University Press},
+  note={\url{https://doi.org/10.1017/S0956796820000088}}
+}
+
+@inproceedings{gueneau2019cycles,
+  title={Formal proof and analysis of an incremental cycle detection algorithm},
+  author={Gu{\'e}neau, Arma{\"e}l and Jourdan, Jacques-Henri and Chargu{\'e}raud, Arthur and Pottier, Fran{\c{c}}ois},
+  booktitle={Interactive Theorem Proving},
+  number={141},
+  year={2019},
+  organization={Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik}
+}
+
+@unpublished{incremental,
+    title = {{Incremental: Library for incremental computations}},
+    author = {{Jane~Street}},
+    year = {2015},
+    note = "\url{https://opensource.janestreet.com/incremental}"
+}
+
+@article{hammer2014adapton,
+  title={Adapton: Composable, demand-driven incremental computation},
+  author={Hammer, Matthew and Phang, Khoo Yit and Hicks, Michael and Foster, Jeffrey},
+  journal={ACM SIGPLAN Notices},
+  volume={49},
+  number={6},
+  pages={156--166},
+  year={2014},
+  publisher={ACM New York, NY, USA}
+}

unikernel/duniverse/dune_/doc/papers/ocaml-2022/refs.csl

@@ -0,0 +1,69 @@
+<?xml version="1.0" encoding="utf-8"?>
+<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0">
+  <info>
+    <title>A bibliography style mostly inspired by ACM-Reference-Format</title>
+    <id>andrey-mokhov-personal-csl</id>
+    <author>
+      <name>Andrey Mokhov</name>
+      <email>andrey.mokhov@gmail.com</email>
+    </author>
+    <category citation-format="author-date"/>
+    <category field="engineering"/>
+    <category field="science"/>
+    <updated>2021-05-20</updated>
+    <rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License</rights>
+  </info>
+  <macro name="author-family-name">
+    <names variable="author">
+      <name form="short" and="text" delimiter=", "/>
+    </names>
+  </macro>
+  <macro name="author-full-name">
+    <names variable="author">
+      <name form="long" and="text" delimiter=", "/>
+    </names>
+  </macro>
+  <macro name="issued-year">
+    <choose>
+      <if variable="issued">
+        <date variable="issued">
+          <date-part name="year"/>
+        </date>
+      </if>
+      <else>
+        <text term="no date"/>
+      </else>
+    </choose>
+  </macro>
+  <citation et-al-min="3" et-al-use-first="1">
+    <sort>
+      <key macro="author-family-name"/>
+      <key macro="issued-year"/>
+    </sort>
+    <layout prefix="[" suffix="]" delimiter="; ">
+      <group delimiter=", ">
+        <text macro="author-family-name"/>
+        <text macro="issued-year"/>
+      </group>
+    </layout>
+  </citation>
+  <bibliography entry-spacing="1">
+    <sort>
+      <key macro="author-full-name"/>
+      <key macro="issued-year"/>
+      <key variable="title" />
+    </sort>
+    <layout>
+      <text macro="author-full-name" suffix=". " />
+      <text variable="title" quotes="true" font-style="italic" suffix="." />
+      <text variable="container-title" prefix=" " suffix="." />
+      <group suffix=".">
+        <text variable="volume" prefix=" vol. " />
+        <text variable="issue" prefix=" (" suffix=")"/>
+        <text variable="page" prefix=", pp. " />
+      </group>
+      <text variable="note" prefix=" " />
+      <text macro="issued-year" prefix=" (" suffix=") " />
+    </layout>
+  </bibliography>
+</style>