unikernel/duniverse/dune_/doc/explanation/preprocessing.rst

@@ -0,0 +1,55 @@
+How Preprocessing Works
+=======================
+
+Preprocessing consists in transforming source code before it is compiled. The
+goal of this document is to explain how this works in Dune.
+
+Dune supports two separate ways of applying preprocessors, the "classic pipeline" (used
+with ``(staged_pps)``), and the "fast pipeline" (used for all other
+:doc:`preprocessing specifications <../reference/preprocessing-spec>` including
+``(pps)``).
+
+The OCaml compilers provide options for specifying a preprocessing step. The
+``-pp`` option is used to invoke a textual preprocessor (something that reads
+text and returns text). The ``-ppx`` option is used to invoke a `ppx rewriter`
+(a function that takes an AST and outputs an AST).
+
+This is the "classic pipeline": preprocessing is part of the compilation
+itself. This is simple, but has a problem: in order to compute the dependencies
+of a module, it is necessary to pass the same ``-pp`` or ``-ppx`` option to
+``ocamldep``.
+
+The classic pipeline has the following steps:
+
+- preprocessing (as part of ``ocamldep``)
+- dependency analysis
+- preprocessing (as part of compilation)
+- compilation
+
+Dune supports a "fast pipeline" where the preprocessor is invoked separately
+from the compiler and its output is saved. Afterwards the preprocessed code is
+compiled directly.
+
+The fast pipeline has the following steps:
+
+- preprocessing
+- dependency analysis
+- compilation
+
+It has several advantages: it only invokes the preprocessor once per file, and
+the preprocessed code is reused between dependency analysis and different kinds
+of compilation. Also, when several preprocessors use ``ppxlib``, they can be
+combined in a preprocessing program that traverses the AST only once.
+
+However, some specific code generators or preprocessors require direct
+access to the compilation artefacts of their dependencies. Therefore they
+need to be used with the classic pipeline, even if it is slower. Note that a
+PPX is able to know if it was called as part of ``ocamldep -ppx`` or ``ocamlopt
+-ppx``, so it can act differently in each phase.
+
+Dune chooses which pipeline to use depending on the
+provided :doc:`../reference/preprocessing-spec`. It will select the fast pipeline,
+unless ``(staged_pps)`` is used. In that case, the classic pipeline is used.
+
+In the case of the fast pipeline, a single executable is built and accepts
+arguments for all preprocessors.