swrup/mte
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Browse source
This commit is contained in:
swrup 2025-11-11 02:07:51 +01:00
parent aa2ff7b2f0
commit 2f3113f55d
11742 changed files with 1223940 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

114
unikernel/duniverse/ppxlib/doc/ast-traversal.mld Normal file
View file

@ -0,0 +1,114 @@
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"matching-code"}< Destructing AST nodes}{%html: </div><div>%}{{!"good-practices"}Good practices >}{%html: </div></div>%}
{0 AST Traversals}
The {{!Ppxlib.Parsetree}[Parsetree]} is a very complex type. Other {!Ppxlib} modules such as
{{!Ppxlib_metaquot}[Metaquot]}, {{!Ppxlib.Ast_builder}[Ast_builder]} and {{!Ppxlib.Ast_pattern}[Ast_pattern]} help in generating and matching values,
but only when the overall structure of the code is known in advance.
For other use cases, such as extracting all identifiers, checking that a
property is verified, or replacing all integer constants by something else,
those modules cannot really help. All these examples relate with another kind
of {{!Ppxlib.Parsetree}[Parsetree]} manipulations known as traversals.
A traversal is a recursive function that will be called on a value, and recursively on all
of its subvalues, combining the result in a certain way. For instance, {{!Stdlib.List.map}[List.map]} is a traversal of the
[list] type. In the case of a [list], a map is very simple to write, but in the
case of the long {{!Ppxlib.Parsetree}[Parsetree]} type, it is a lot of boilerplate code! Fortunately,
{{!Ppxlib}[ppxlib]} provides a way to ease this.
In [ppxlib], traversals are implemented using the "visitor" object-oriented pattern.
{1 Writing Traverses}
For each kind of traversal (described below), [ppxlib] provides a "default" traversal,
in the form of a class following the visitors pattern. For instance, in the case of the map traversal, the
default map is the identity AST map, and any object of class {{!Ppxlib.Ast_traverse.map}[Ast_traverse.map]}
will be this identity map. To apply a map to a node of a given type, one needs
to call the appropriate method:
{[
# let f payload =
let map = new Ppxlib.Ast_traverse.map in
map#payload ;;
val f : payload -> payload = <fun>
]}
In the example above, [f] is the identity map. But we want to define proper maps,
not just identity. This is done by creating a new class, making it inherit the
methods, and replacing the one that we want to replace. Here is an example, for
both the [iter] and [map] traversals:
{[
let f payload =
let checker =
object
inherit Ast_traverse.iter as super
method! extension ext =
match ext with
| { txt = "forbidden"; _ }, _ ->
failwith "Fordidden extension nodes are forbidden!"
| _ -> super#extension ext (* Continue traversing inside the node *)
end
in
let replace_constant =
object
inherit Ast_traverse.map
method! int i = i + 1
end
in
checker#payload payload;
replace_constant#payload payload
]}
Note that when redefining methods, unless explicitly wanting the traversal to
stop, the original method needs to be called! That should be all that’s necessary to
know and understand the {{!Ppxlib.Ast_traverse.map}API}.
{1 The Different Kinds of Traversals}
{{!Ppxlib}[ppxlib]} offers different kind of {{!Ppxlib.Parsetree}[Parsetree]} traversals:
- {{!Ppxlib.Ast_traverse.iter}Iterators}, which will traverse the type, calling
a function on each node for side effects.
- {{!Ppxlib.Ast_traverse.map}Maps}, where the content is replaced. A map will
transform a [Parsetree] into another [Parsetree], replacing nodes following the
map function.
- {{!Ppxlib.Ast_traverse.fold}Folds}, which will traverse the nodes, carrying a
value (often called an accumulator) that will be updated on each node.
- {{!Ppxlib.Ast_traverse.lift}Lifts}, a transformation that turns a [Parsetree] value in one of another type by
transforming it in a bottom-up manner. For instance, with a simple tree
structure, the corresponding [lift] function would be:
{[
let lift ~f = function
Leaf a -> f.leaf a
| Node(a,x,y) -> f.node a (lift ~f x) (lift ~f y)
]}
- Combinations of the two traversals, such as
{{!Ppxlib.Ast_traverse.fold_map}Fold-maps} and
{{!Ppxlib.Ast_traverse.lift_map_with_context}Lift-maps}.
- Variants of the above traversal, such as
{{!Ppxlib.Ast_traverse.map_with_context}Maps with context}, where a context
can be modified and passed down to child nodes during traversal. The context
never goes up; it is only propagated down. It is used for instance to track
opened module. To give a simple example, such a context could be the depth of
the current node, as in the following implementation for the simple tree type:
{[
let map_with_depth_context ~f ctxt = function
Leaf a -> f.leaf ctxt a
| Node(a,x,y) ->
f.node ctxt a
(map_with_depth_context (ctxt+1) ~f x)
(map_with_depth_context (ctxt+1) ~f y)
]}
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"matching-code"}< Destructing AST nodes}{%html: </div><div>%}{{!"good-practices"}Good practices >}{%html: </div></div>%}

413
unikernel/duniverse/ppxlib/doc/driver.mld Normal file
View file

@ -0,0 +1,413 @@
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"quick_intro"}< Introduction}{%html: </div><div>%}{{!"writing-ppxs"}Writing PPXs >}{%html: </div></div>%}
{0 How It Works}
{1 General Concepts}
{2 The Driver}
[ppxlib] sits in between the PPXs authors and the compiler toolchain. For the PPX
author, it provides an API to define the transformation and register it to
[ppxlib]. Then, all registered transformations can be turned into a single
executable, called the {e driver}, that is responsible for applying all the
transformations. The driver will be called by the compiler.
The PPX authors register their transformations using the
{{!Ppxlib.Driver.register_transformation}[Driver.register_transformation]} function, as explained in the
{{!"writing-ppxs"}Writing PPXs} section. The different arguments of this function
corresponds to the {{!"derivers-and-extenders"}different kinds} of PPXs supported by
[ppxlib] or the {{!driver_execution}phase}, at which time they will be executed.
The driver is created by calling either {{!Ppxlib.Driver.standalone}[Driver.standalone]} or
{{!Ppxlib.Driver.run_as_ppx_rewriter}[Driver.run_as_ppx_rewriter]}. Note that when used through Dune, none of
these functions will need to be called by the PPX author. As we will see, Dune
will be responsible for generating the driver after all required PPXs from
different libraries have been registered. These functions will interpret the
command line arguments and start the rewriting accordingly.
The {{!Ppxlib.Driver.standalone}[Driver.standalone]} function creates an executable that
parses an OCaml file, transforms it according to the registered transformations,
and outputs the transformed file. This makes it suitable for use with the [-pp]
{{:https://v2.ocaml.org/releases/5.0/htmlman/comp.html#s:comp-options}option} of the OCaml compiler. It is a preprocessor for sources and is
standalone in the sense that it can be called independently from the OCaml
compiler (e.g., it includes an OCaml parser).
On the other hand, the
{{!Ppxlib.Driver.run_as_ppx_rewriter}[Driver.run_as_ppx_rewriter]}-generated driver is a
proper PPX, as it will read and output a {{!Ppxlib.Parsetree}[Parsetree]} marshalled
value directly. This version is suitable for use with the [-ppx] {{:https://v2.ocaml.org/releases/5.0/htmlman/comp.html#s:comp-options}option} of the OCaml
compiler, as well as any tool that requires control of parsing the file.
For instance, {{:https://ocaml.github.io/merlin/}Merlin} includes an OCaml parser that tries
hard to recover from errors in order to generate a valid AST most of the time.
Several arguments can be passed to the driver when executing it. Those arguments
can also be easily passed using Dune, as explained in its
{{:https://dune.readthedocs.io/en/stable/concepts.html#preprocessing-with-ppx-rewriters}manual}.
PPX authors can add arguments to their generated drivers using {{!Ppxlib.Driver.add_arg}[Driver.add_arg]}. Here are the default arguments for respectively
{{!Ppxlib.Driver.standalone}[standalone]} and
{{!Ppxlib.Driver.run_as_ppx_rewriter}[run_as_ppx_rewriter]} generated drivers:
{%html: <details><summary>Standalone driver</summary>%}
{v
driver.exe [extra_args] [<files>]
-as-ppx Run as a -ppx rewriter (must be the first argument)
--as-ppx Same as -as-ppx
-as-pp Shorthand for: -dump-ast -embed-errors
--as-pp Same as -as-pp
-o <filename> Output file (use '-' for stdout)
- Read input from stdin
-dump-ast Dump the marshaled ast to the output file instead of pretty-printing it
--dump-ast Same as -dump-ast
-dparsetree Print the parsetree (same as ocamlc -dparsetree)
-embed-errors Embed errors in the output AST (default: true when -dump-ast, false otherwise)
-null Produce no output, except for errors
-impl <file> Treat the input as a .ml file
--impl <file> Same as -impl
-intf <file> Treat the input as a .mli file
--intf <file> Same as -intf
-debug-attribute-drop Debug attribute dropping
-print-transformations Print linked-in code transformations, in the order they are applied
-print-passes Print the actual passes over the whole AST in the order they are applied
-ite-check (no effect -- kept for compatibility)
-pp <command> Pipe sources through preprocessor <command> (incompatible with -as-ppx)
-reconcile (WIP) Pretty print the output using a mix of the input source and the generated code
-reconcile-with-comments (WIP) same as -reconcile but uses comments to enclose the generated code
-no-color Don't use colors when printing errors
-diff-cmd Diff command when using code expectations (use - to disable diffing)
-pretty Instruct code generators to improve the prettiness of the generated code
-styler Code styler
-output-metadata FILE Where to store the output metadata
-corrected-suffix SUFFIX Suffix to append to corrected files
-loc-filename <string> File name to use in locations
-reserve-namespace <string> Mark the given namespace as reserved
-no-check Disable checks (unsafe)
-check Enable checks
-no-check-on-extensions Disable checks on extension point only
-check-on-extensions Enable checks on extension point only
-no-locations-check Disable locations check only
-locations-check Enable locations check only
-apply <names> Apply these transformations in order (comma-separated list)
-dont-apply <names> Exclude these transformations
-no-merge Do not merge context free transformations (better for debugging rewriters). As a result, the context-free transformations are not all applied before all impl and intf.
-cookie NAME=EXPR Set the cookie NAME to EXPR
--cookie Same as -cookie
-deriving-keep-w32 {impl|intf|both}
Do not try to disable warning 32 for the generated code
-deriving-disable-w32-method {code|attribute}
How to disable warning 32 for the generated code
-type-conv-keep-w32 {impl|intf|both}
Deprecated, use -deriving-keep-w32
-type-conv-w32 {code|attribute}
Deprecated, use -deriving-disable-w32-method
-deriving-keep-w60 {impl|intf|both}
Do not try to disable warning 60 for the generated code
-unused-code-warnings {true|false|force}
Allow ppx derivers to enable unused code warnings (default: false)
-unused-type-warnings {true|false|force}
Allow unused type warnings for types with [@@deriving ...] (default: false)
-help Display this list of options
--help Display this list of options
v}
{%html: </details>%}
and
{%html: <details><summary>Ppx rewriter driver</summary>%}
{v
driver.exe [extra_args] <infile> <outfile>
-loc-filename <string> File name to use in locations
-reserve-namespace <string> Mark the given namespace as reserved
-no-check Disable checks (unsafe)
-check Enable checks
-no-check-on-extensions Disable checks on extension point only
-check-on-extensions Enable checks on extension point only
-no-locations-check Disable locations check only
-locations-check Enable locations check only
-apply <names> Apply these transformations in order (comma-separated list)
-dont-apply <names> Exclude these transformations
-no-merge Do not merge context free transformations (better for debugging rewriters). As a result, the context-free transformations are not all applied before all impl and intf.
-cookie NAME=EXPR Set the cookie NAME to EXPR
--cookie Same as -cookie
-help Display this list of options
--help Display this list of options
v}
{%html: </details>%}
{3:exception_handling Exception handling}
In general, raising an exception in a registered transformation will make the
ppxlib driver crash with an uncaught exception error. However, when spawned with
the [-embed-errors] or [-as-ppx] flags (that's the case when Merlin calls the
driver) the ppxlib driver still handles a specific kind of exception: Located
exceptions. They have type {{!Ppxlib.Location.exception-Error}[Location.Error]}
and contain enough information to display a located error message.
During its {{!page-driver.driver_execution}execution}, the driver will run many
different rewriters. In the case described above, it will catch any located
exception thrown by a rewriter. When catching an exception, it will collect the
error in a list, take the last valid AST (the one that was given to the raising
rewriter) and continue its execution from there.
At the end of the rewriting process, the driver will prepend all collected errors
to the beginning of the AST, in the order in which they appeared.
The same mechanism applies for the {{!"derivers-and-extenders"}context-free rewriters}: if any of them raises,
the error is collected, the part of the AST that the rewriter was responsible to
rewrite remains unmodified, and the {{!"context-free-phase"}context-free phase} continues.
{2 Cookies}
Cookies are values that are passed to the driver via the command line, or set as
side effects of transformations, which can be accessed by the
transformations. They have a name to identify them and a value consisting of an
OCaml expression. The module to access cookies is {{!Ppxlib.Driver.Cookies}[Driver.Cookies]}.
{2 Integration With Dune}
The {{:https://dune.build}Dune} build system is well integrated with the [ppxlib]
mechanism of registering transformations. In every [dune] file, Dune will read
the set of PPXs that are to be used (i.e. the PPXs in `(preprocess (pps <list of PPXs that are to be used>))`). For a given set of rewriters, it will
generate a driver using {{!Ppxlib.Driver.run_as_ppx_rewriter}[Driver.run_as_ppx_rewriter]} that contains all registered
transformations. Using a single driver for multiple
transformations from multiple PPXs ensures better composition semantics and
improves the speed of the combined transformations.
Moreover, [ppxlib] communicates with Dune through [.corrected] files to allow for
promotion, for instance when using {{!"writing-ppxs"."inlining-transformations"}[[@@deriving_inline]]}. A PPX author can also
generate its own promotion suggestion using the
{{!Ppxlib.Driver.register_correction}[Driver.register_correction]} function.
{1:compat_mult_ver Compatibility With Multiple OCaml Versions}
One of the important issues with working with the
{{!Ppxlib.Parsetree}[Parsetree]} is that the API is not stable. For instance, in
the {{:https://ocaml.org/releases/4.13.0}OCaml 4.13 release}, the following
{{:https://github.com/ocaml/ocaml/pull/9584/files#diff-ebecf307cba2d756cc28f0ec614dfc57d3adc6946eb4faa9825eb25a92b2596d}two}
{{:https://github.com/ocaml/ocaml/pull/10133/files#diff-ebecf307cba2d756cc28f0ec614dfc57d3adc6946eb4faa9825eb25a92b2596d}changes}
were made to the {{!Ppxlib.Parsetree}[Parsetree]} type. Although they are small changes, they may
break any PPX that is written to directly manipulate the (evolving) type.
This instability causes an issue with maintenance. PPX authors wish to maintain
a single version of their PPX, not one per OCaml version, and ideally not have
to update their code when an irrelevant (for them) field is changed in the
{{!Ppxlib.Parsetree}[Parsetree]}.
[ppxlib] helps to solve both issues. The first one, having to maintain a single
PPX version working for every OCaml version, is done by migrating the
{{!Ppxlib.Parsetree}[Parsetree]}. The PPX author only maintains a version
working with the latest version, and the [ppxlib] driver will convert the values from one version to another.
For example, say a deriver is applied in the context of OCaml 4.08. After the
4.08 {{!Ppxlib.Parsetree}[Parsetree]} has been given to it, the [ppxlib] driver
will migrate this value into the latest {{!Ppxlib.Parsetree}[Parsetree]}
version, using the {!Astlib} module. The "latest" here depends on the version of
[ppxlib], but at any given time, the latest released version of [ppxlib] will always
use the latest released version of the {{!Ppxlib.Parsetree}[Parsetree]}.
After the migration to the latest {{!Ppxlib.Parsetree}[Parsetree]},
the driver runs all transformations on it, which ends with a rewritten
{{!Ppxlib.Parsetree}[Parsetree]} of the latest version. However, since the
context of rewriting is OCaml 4.08 (in this example), the driver needs to
migrate back the rewritten {{!Ppxlib.Parsetree}[Parsetree]} to an OCaml 4.08
version. Again, [ppxlib] uses the {!Astlib} module for this migration. Once the
driver has rewritten the AST for OCaml 4.08, the compilation can continue as usual.
{1:derivers-and-extenders Context-Free Transformations}
[ppxlib] defines several kinds of transformations whose core property is that they
can only read and modify the code locally. The parts of the AST given
to the transformation are only portions of the whole AST. In this regard, they
are usually called {e context-free} transformations. While being not as
general-purpose as plain AST transformations, they are more than often
sufficient and have many nice properties such as a well-defined semantics for
composition.
The two most important context-free transformations are {e derivers} and
{e extenders}.
{2:def_derivers Derivers}
A {e deriver} is a context-free transformation that, given a certain structure or
signature item, will generate code {e to append after} this item. The given code is
never modified. A deriver can be very useful to generate values depending on the
structure of a user-defined type, for instance a converter for a type
to and from a JSON value. A deriver is triggered by adding an
{{:https://v2.ocaml.org/manual/attributes.html}attribute} to a structure or
signature item. For instance, the folowing code:
{@ocaml[
type t = Int of int | Float of float [@@deriving yojson]
let x = ...
]}
would be rewritten to:
{@ocaml[
type ty = Int of int | Float of float [@@deriving yojson]
let ty_of_yojson = ...
let ty_to_yojson = ...
let x = ...
]}
{2:def_extenders Extenders}
An {e extender} is a context-free transformation that is triggered on
{{:https://v2.ocaml.org/manual/extensionnodes.html}extension nodes}, and that
will replace the extension node by some code generated from the extension node's payload. This can be very useful to generate values of a DSL using a more
user-friendly syntax, e.g., to generate OCaml values from the JSON
syntax.
For instance, the following code:
{@ocaml[
let json =
[%yojson
[ { name = "Anne"; grades = ["A"; "B-"; "B+"] }
; { name = "Bernard"; grades = ["B+"; "A"; "B-"] }
]
]
]}
could be rewritten into:
{@ocaml[
let json =
`List
[ `Assoc
[ ("name", `String "Anne")
; ("grades", `List [`String "A"; `String "B-"; `String "B+"])
]
; `Assoc
[ ("name", `String "Bernard")
; ("grades", `List [`String "B+"; `String "A"; `String "B-"])
]
]
]}
{2 Advantages}
There are multiple advantages of using context-free transformations. First, they
provide the PPX user a much clearer understanding of the AST parts that
will be rewritten, rather than a fully general AST rewriting. Secondly, they provide
a much better composition semantic, which does not depend on the order. Finally,
context-free transformations are applied in a single phase factorising the work
for all transformations, resulting in a much faster driver than when combining
multiple, whole AST transformations. More details on the execution of this phase
are given in its {{!"context-free-phase"}dedicated section.}
See the {{!"writing-ppxs"}Writing PPXs} section for how to define derivers and
extenders.
{1:driver_execution The Execution of the Driver}
The actual rewriting of the AST is done in multiple phases:
{ol
{- The linting phase}
{- The preprocessing phase}
{- The first instrumentation phase}
{- The context-free phase}
{- The global transformation phase}
{- The last instrumentation phase}
}
When registering a transformation through the
{{!Ppxlib.Driver.register_transformation}[Driver.register_transformation]} function, the phase in which the
transformation has to be applied is specified. The multiplicity of phases is
mostly to account for potential constraints on the execution order. However,
most of the time there are no such constraints, and in this case, either the
{{!"context-free-phase"}context-free} or the
{{!"global-transfo-phase"}global transformation phase} should be used. (Note that
whenever possible, which should be almost always, context-free transformations
are possible and better.) If you register in another phase, be sure to know what
you are doing.
{2 The Linter Phase}
Linters are preprocessors that take as input the whole AST and output a list
of "lint" errors. Such an error is of type {{!Ppxlib.Driver.Lint_error.t}[Driver.Lint_error.t]} and includes a
string (the error message) and the location of the error. The errors will be
reported as preprocessors warnings.
This is the first phase, so linting errors can only be reported for code
handwritten by the user.
An example of a PPX registered in this phase is
{{:https://github.com/janestreet/ppx_js_style}ppx_js_style}.
{2 The Preprocessing Phase}
The preprocessing phase is the first transformation that actually alters the
AST. In fact, the property of being the "first transformation applied" is what
defines this phase, and [ppxlib] will thus ensure that only one transformation is
registered in this phase; otherwise, it will generate an error.
You should only register a transformation in this phase if it is really strongly
necessary, and you know what you are doing. Your PPX will not be usable at the
same time as another one registering a transformation in this phase.
An example of a PPX registered in this phase is
{{:https://github.com/thierry-martinez/metapp}metapp}.
{2 The First Instrumentation Phase}
This phase is for transformations that {e need} to be run before the
context-free phase. Historically, it was meant for
{{:https://en.wikipedia.org/wiki/Instrumentation_(computer_programming)}instrumentation}-related
PPXs, hence the name. Unlike the {{!"the-preprocessing-phase"}preprocessing
phase}, registering to this phase provides no guarantee that the transformation
is run early in the rewriting, as there is no limit in the number of
transformations registered in this phase, which are then applied in the
alphabetical order by their name.
If it is not crucial for a transformation to run before the context-free
phase, it should be registered to the {{!"global-transfo-phase"}global
transformation phase}.
{2:context-free-phase The Context-Free Phase}
The execution of all registered context-free rules is done in a single top-down
pass through the AST. Whenever the top-down pass encounters a situation
that triggers rewriting, the corresponding transformation is called. For instance,
when encountering an extension point corresponding to a rewriting rule, the
extension point is replaced by the rule's execution, and the top-down pass
continues inside the generated code. Similarly, when a deriving attribute is
found attached to a structure or signature item, the result of the
deriving rule’s application is appended to the AST, and the top-down pass
continues in the generated code.
Note that the code generation for derivers is applied when "leaving" the AST
node, that is when all rewriters have been run. Indeed, a deriver like this:
{[
type t = [%my_type] [@@deriving deriver_from_type]
]}
would need the information generated by the [my_type] extender to match on the
structure of [t].
Also note that in this phase, the execution of the context-free rules are
intertwined altogether, and it would not make sense to speak about the order of
application, contrary to the next phase.
{2:global-transfo-phase The Global Transformation Phase}
The global transformation phase is the phase where registered transformations,
seen as function from and to the {{!Ppxlib.Parsetree}[Parsetree]}, are run. The applied order might matter and change the outcome, but since [ppxlib] knows nothing
about the transformations, the order applied is alphabetical by the transformation's name.
{2 The Last Instrumentation Phase}
This phase is for global transformation to escape the alphabetical order and be
executed as a last phase. For instance, {{:https://github.com/aantron/bisect_ppx}[bisect_ppx]}
needs to be executed after all rewriting has occurred.
Note that only one global transformation can be executed last. If several
transformations rely on being the last transformation, it will be true for only
one of them. Thus, only register your transformation in this phase if it is
absolutely vital to be the last transformation, as your PPX will become
incompatible with any other that registers a transformation during this phase.
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"quick_intro"}< Introduction}{%html: </div><div>%}{{!"writing-ppxs"}Writing PPXs >}{%html: </div></div>%}

2
unikernel/duniverse/ppxlib/doc/dune Normal file
View file

@ -0,0 +1,2 @@
(documentation
(package ppxlib))

167
unikernel/duniverse/ppxlib/doc/examples.mld Normal file
View file

@ -0,0 +1,167 @@
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"good-practices"}< Good practices}{%html: </div><div>%}{%html: </div></div>%}
{0 Examples}
This section is here to allow viewing complete examples of PPXs written using [ppxlib] directly in the documentation. However, they are not "complete" in the sense that the overall organization, such as the [dune] files, is not included.
In order to see a fully working complete example of a PPX written using [ppxlib], that you can compile, modify and test, go to the {{:https://github.com/ocaml-ppx/ppxlib/tree/main/examples}examples} folder of ppxlib sources.
{1 [ppx_deriving_accesors]}
The fully complete, ready-to-compile [ppx_deriving_accesors] example is accessible in [ppxlib]'s {{:https://github.com/ocaml-ppx/ppxlib/tree/main/examples/simple-deriver}sources}.
This deriver will generate accessors for record fields, from the record type
definition.
For example, this code:
{@ocaml[
type t =
{ a : string
; b : int
}
[@@deriving accessors]
]}
will generate the following, appended after the type definition:
{@ocaml[
let a x = x.a
let b x = x.b
]}
The entire code is:
{@ocaml[
open Ppxlib
module List = ListLabels
open Ast_builder.Default
let accessor_impl (ld : label_declaration) =
let loc = ld.pld_loc in
pstr_value ~loc Nonrecursive
[
{
pvb_pat = ppat_var ~loc ld.pld_name;
pvb_expr =
pexp_fun ~loc Nolabel None
(ppat_var ~loc { loc; txt = "x" })
(pexp_field ~loc
(pexp_ident ~loc { loc; txt = lident "x" })
{ loc; txt = lident ld.pld_name.txt });
pvb_attributes = [];
pvb_loc = loc;
};
]
let accessor_intf ~ptype_name (ld : label_declaration) =
let loc = ld.pld_loc in
psig_value ~loc
{
pval_name = ld.pld_name;
pval_type =
ptyp_arrow ~loc Nolabel
(ptyp_constr ~loc { loc; txt = lident ptype_name.txt } [])
ld.pld_type;
pval_attributes = [];
pval_loc = loc;
pval_prim = [];
}
let generate_impl ~ctxt (_rec_flag, type_declarations) =
let loc = Expansion_context.Deriver.derived_item_loc ctxt in
List.map type_declarations ~f:(fun (td : type_declaration) ->
match td with
| {
ptype_kind = Ptype_abstract | Ptype_variant _ | Ptype_open;
ptype_loc;
_;
} ->
let ext =
Location.error_extensionf ~loc:ptype_loc
"Cannot derive accessors for non record types"
in
[ Ast_builder.Default.pstr_extension ~loc ext [] ]
| { ptype_kind = Ptype_record fields; _ } ->
List.map fields ~f:accessor_impl)
|> List.concat
let generate_intf ~ctxt (_rec_flag, type_declarations) =
let loc = Expansion_context.Deriver.derived_item_loc ctxt in
List.map type_declarations ~f:(fun (td : type_declaration) ->
match td with
| {
ptype_kind = Ptype_abstract | Ptype_variant _ | Ptype_open;
ptype_loc;
_;
} ->
let ext =
Location.error_extensionf ~loc:ptype_loc
"Cannot derive accessors for non record types"
in
[ Ast_builder.Default.psig_extension ~loc ext [] ]
| { ptype_kind = Ptype_record fields; ptype_name; _ } ->
List.map fields ~f:(accessor_intf ~ptype_name))
|> List.concat
let impl_generator = Deriving.Generator.V2.make_noarg generate_impl
let intf_generator = Deriving.Generator.V2.make_noarg generate_intf
let my_deriver =
Deriving.add "accessors" ~str_type_decl:impl_generator
~sig_type_decl:intf_generator
]}
{1 [ppx_get_env]}
The fully complete, ready-to-compile [ppx_get_env] example is accessible in [ppxlib]'s {{:https://github.com/ocaml-ppx/ppxlib/tree/main/examples/simple-extension-rewriter}sources}.
A PPX rewriter that will expand [[%get_env "SOME_ENV_VAR"]] into the value of the
env variable [SOME_ENV_VAR] at compile time, as a string.
E.g., assuming we set [MY_VAR="foo"], it will turn:
{@ocaml[
let () = print_string [%get_env "foo"]
]}```
into:
{@ocaml[
let () = print_string "foo"
]}
Note that this is just a toy example, and we actually advise against this
type of PPX that has side effects or relies heavily on the file system or [env]
variables, unless you absolutely you know what you're doing.
In this case, it won't work well with Dune, since Dune won't know
about the dependency on the env variables specified in the extension's payload.
The entire code is:
{@ocaml[
open Ppxlib
let expand ~ctxt env_var =
let loc = Expansion_context.Extension.extension_point_loc ctxt in
match Sys.getenv env_var with
| value -> Ast_builder.Default.estring ~loc value
| exception Not_found ->
let ext =
Location.error_extensionf ~loc "The environment variable %s is unbound"
env_var
in
Ast_builder.Default.pexp_extension ~loc ext
let my_extension =
Extension.V3.declare "get_env" Extension.Context.expression
Ast_pattern.(single_expr_payload (estring __))
expand
let rule = Ppxlib.Context_free.Rule.extension my_extension
let () = Driver.register_transformation ~rules:[ rule ] "get_env"
]}
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"good-practices"}< Good practices}{%html: </div><div>%}{%html: </div></div>%}

360
unikernel/duniverse/ppxlib/doc/generating-code.mld Normal file
View file

@ -0,0 +1,360 @@
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"writing-ppxs"}< Writing PPXs}{%html: </div><div>%}{{!"matching-code"}Destructing AST nodes >}{%html: </div></div>%}
{0 Generating AST Nodes}
The rewriter's core is a function that outputs code in the form of an AST.
However, there are some issues with generating AST values when using the
constructors directly:
- The type is {{!Ppxlib.Parsetree}pretty verbose}, with many fields
rarely used.
- The AST type might change at a version bump. In this case, the types used in
the PPX would become incompatible with the types of the new OCaml version.
The second point is important: since [ppxlib] {{!page-driver.compat_mult_ver}translates} the AST to
the newest OCaml AST available before rewriting, your PPX would not
only become incompatible with the new OCaml version, but also with all [ppxlib]
versions released after the new AST type is introduced.
For this reason, [ppxlib] provides abstractions over the OCaml AST, with a focus
on usability and stability.
{1 The Different Options}
The two main options are:
- {{!Ppxlib.Ast_builder}[Ast_builder]},
- {!Ppxlib_metaquot}.
{{!Ppxlib.Ast_builder}[Ast_builder]} provides an API to generate AST nodes for the latest OCaml
version in a backward-compatible way. {!Ppxlib_metaquot} is different: it is a PPX that
lets you generate OCaml AST nodes by writing OCaml code, using quotations and
anti-quotations.
Using {!Ppxlib_metaquot} requires less knowledge of the OCaml AST than
{{!Ppxlib.Ast_builder}[Ast_builder]} as it only uses natural OCaml syntax;
however, it's more restrictive than `Ast_builder` for two reasons: first, it's less flexible, since on its own it lacks the ability to generate nodes dynamically from other kind of data: e.g. it's not possible to build an expression containing a string, given the string as input. Second, it's less general because it only allows users to generate few different nodes such as
structure items, expressions, patterns, etc., but it is not possible to generate a
value of type {{!Ppxlib.Parsetree.row_field_desc}[row_field_desc]}! A typical workflow is to use `metaquot` for the constant skeleton of the node, and to use the `metaquot` anti-quotation workflow (see below) together with `Ast_builder` to fill in the dynamic parts.
Note: `Ppxlib` also re-exports the OCaml compiler API `Ast_helper` for historic reasons. It might get deprecated at some point, though. Please, use `Ast_builder` instead.
manipulate the AST. This module is in [ppxlib] for compatiblity reasons and it is recommended to use {{!Ppxlib.Ast_builder}[Ast_builder]} instead.
{1:ast_builder The [AST_builder] Module}
{2 General Presentation}
The {{!Ppxlib.Ast_builder}[Ast_builder]} module provides several kinds of functions to generate AST
nodes. The first kind are ones whose name matches closely the {{!Ppxlib.Parsetree}[Parsetree]} type names.
equivalents, but there are also "higher level" wrappers around those basic
blocks for common patterns such as creating an integer or string constant.
{3 Low-Level Builders}
The function names match the {{!Ppxlib.Parsetree}[Parsetree]} names closely, which makes it easy to
build AST fragments by just knowing the {{!Ppxlib.Parsetree}[Parsetree]}.
For types wrapped in a record's [_desc] field, helpers are
generated for each constructor that generates the record wrapper, e.g.,
for the type {{!Ppxlib.Parsetree.expression}[Parsetree.expression]}:
{[
type expression =
{ pexp_desc : expression_desc
; pexp_loc : Location.t
; pexp_attributes : attributes
}
and expression_desc =
| Pexp_ident of Longident.t loc
| Pexp_constant of constant
| Pexp_let of rec_flag * value_binding list * expression
...
]}
The following helpers are created:
{[
val pexp_ident : loc:Location.t -> Longident.t loc -> expression
val pexp_constant : loc:Location.t -> constant -> expression
val pexp_let : loc:Location.t -> rec_flag -> value_binding list -> expression -> expression
...
]}
For other record types, such as [type_declaration], we have the following
helper:
{[
type type_declaration =
{ ptype_name : string Located.t
; ptype_params : (core_type * variance) list
; ptype_cstrs : (core_type * core_type * Location.t) list
; ptype_kind : type_kind
; ptype_private : private_flag
; ptype_manifest : core_type option
; ptype_attributes : attributes
; ptype_loc : Location.t
}
val type_declaration
: loc : Location.t
-> name : string Located.t
-> params : (core_type * variance) list
-> cstrs : (core_type * core_type * Location.t) list
-> kind : type_kind
-> private : private_flag
-> manifest : core_type option
-> type_declaration
]}
Attributes are always set to the empty list. If you want to set them, you
have to override the field with the [{ e with pexp_attributes = ... }]
notation.
{3 High-Level Builders}
Those functions are just wrappers on the low-level functions for simplifying the
most common use. For instance, to simply create a [1] integer constant with the
low-level building block, it would look like:
{[
Ast_builder.Default.pexp_constant ~loc (Parsetree.Pconst_integer ("1", None))
]}
This seems a lot for such a simple node. So, in addition to the low-level
building blocks, {{!Ppxlib.Ast_builder}[Ast_builder]} provides higher level-building blocks, such as
{{!Ppxlib.Ast_builder.Default.eint}[Ast_builder.Default.eint]}, to create integer constants:
{[
Ast_builder.Default.eint ~loc 1
]}
Those functions also follow a pattern in their name to make them easier to use.
Functions that generate an expression start with an [e], followed by what they
build, such as [eint], [echar], [estring], [eapply], [elist], etc. Similarly, names
that start with a [p] define a pattern, such as [pstring], [pconstruct],
[punit], etc.
{2 Dealing With Locations}
As explained in the {{!page-"good-practices"."resp_loc"}dedicated section}, it is crucial
to correctly deal with locations. For this, {{!Ppxlib.Ast_builder}[Ast_builder]} can be used in
several ways, depending on the context:
{{!Ppxlib.Ast_builder.Default}[Ast_builder.Default]} contains functions which take the location as a named
argument. This is the strongly recommended workflow and lets you control locations in a fine-grained way.
If you have a concrete reason to specify the location once and for all, and always use this
specific one later in AST constructions, you can use the {{!Ppxlib.Ast_builder.Make}[Ast_builder.Make]} functor
or the {{!Ppxlib.Ast_builder.make}[Ast_builder.make]} function (outputing a first order module). Notice that this is quite a rare use case.
{2 Compatibility}
In order to stay as compatible as possible when a new option appears in the AST,
{{!Ppxlib.Ast_builder}[Ast_builder]} always integrates the new option in a retro-compatible way (this is the case since the AST bump from 4.13 to 4.14). So, the
signature of each function won't change, and {{!Ppxlib.Ast_builder}[Ast_builder]} will choose a
retrocompatible way of generating an updated type’s AST node.
However, sometimes you might want to use a feature that was introduced recently
in OCaml and is not integrated in {{!Ppxlib.Ast_builder}[Ast_builder]}. For instance, OCaml
4.14 introduced the possibility to explicitly introduce type variables in a
constructor declaration. This modified the AST type, and for
backwards compatibility, {{!Ppxlib.Ast_builder}[Ast_builder]} did not modify the signature of the
function. It is thus impossible to generate code using this new feature via the `Ast_module` directly.
In the case you need to access a new feature, you can use the [Latest] submodule
(e.g., {{!Ppxlib.Ast_builder.Default.Latest}[Ast_builder.Default.Latest]} when specifying the locations). This module includes new functions, letting you
control all features introduced, at the cost of potentially breaking
changes when a new feature modifies the function in use.
If a feature that was introduced in some recent version of OCaml is essential
for your PPX to work, it might imply that you need to restrict the OCaml version
on your opam dependencies.
{{!page-driver.compat_mult_ver}Remember} that
[ppxlib] will rewrite using the latest [Parsetree] version, {e but} it will then migrate the
[Parsetree] back to the OCaml version of the switch, possibly losing the information
given by the new feature.
{1:metaquot [Metaquot] Metaprogramming}
{2 General Presentation}
As you have seen, defining code with {{!Ppxlib.Ast_builder}[Ast_builder]} does
not feel perfectly natural. Some knowledge of the [Parsetree] types is needed.
Yet, every part of a program we write corresponds to a specific AST node, so
there is no need for AST generation to be more difficult than that.
[Metaquot] is a very useful PPX that allows users to define values of a [Parsetree]
type by writing natural code, using the quotations and antiquotations mechanism of
metaprogramming.
Simplifying a bit, {{!Ppxlib_metaquot}[Metaquot]} rewrites an expression extension point directly
with its payload. Since the payload was parsed by the OCaml parser to a
[Parsetree] type's value, this rewriting turns naturally written code into AST values.
{2 Usage}
First, in order to use [Metaquot], add it in your [preprocess] Dune stanza:
{[
(preprocess (pps ppxlib.metaquot))
]}
Using Metaquot to generate code is simple: any [Metaquot] extension node in an
expression context will be rewritten into the [Parsetree] value that lies in its payload.
Notice that you'll need the [Ppxlib] opened, and a [loc] value of type
{{!Ppxlib.Location.t}Location.t} in scope when using metaquot. That location
will be attached to the [Parsetree] nodes your metaquot invokation produces.
Getting the location right is extremely important for error messages.
However, the {{!Ppxlib.Parsetree.payload}[Parsetree.payload]} of an extension node can only take few forms: a
{{!Ppxlib.Parsetree.structure}[structure]}, a {{!Ppxlib.Parsetree.signature}[signature]}, a {{!Ppxlib.Parsetree.core_type}[core type]}, or a {{!Ppxlib.Parsetree.pattern}[pattern]}. We might want to generate
other kind of nodes, such as {{!Ppxlib.Parsetree.expression}[expressions]} or {{!Ppxlib.Parsetree.structure_item}[structure items]}, for instance.
{!Ppxlib_metaquot} provides different extension nodes for this:
- The [expr] extension node to generate {{!Ppxlib.Parsetree.expression}[expressions]}:
{[let e = [%expr 1 + 1]]}
- The [pat] extension node to generate {{!Ppxlib.Parsetree.pattern}[patterns]}:
{[let p = [%pat? ("", _)]]}
- The [type] extension node to generate {{!Ppxlib.Parsetree.core_type}[core types]}:
{[let t = [%type: int -> string]]}
- The [stri] extension node to generate {{!Ppxlib.Parsetree.structure_item}[structure_item]}, with its [sigi] counterpart for {{!Ppxlib.Parsetree.signature_item}[signature_item]}::
{[
let stri = [%stri let a = 1]
let sigi = [%sigi: val i : int]
]}
- The [str] and [sig] extension nodes to respectively generate
{{!Ppxlib.Parsetree.structure}[structure]}
and {{!Ppxlib.Parsetree.signature}[signature]}.
{[
let str =
[%str
let x = 5
let y = 6.3]
let sig_ =
[%sig:
val x : int
val y : float]
]}
Note the replacement work when the extension node is an "expression"
extension node: Indeed, the [payload] is a {e value} (of [Parsetree] type) that would not fit
elsewhere in the AST. So, [let x : [%str "incoherent"]] would not be rewritten by [metaquot].
(Actually, it also rewrites "pattern" extension nodes, as you'll see in the
chapter on {{!page-"matching-code".metaquot}matching AST nodes}.)
Also note the [:] and [?] in the [sigi], [type], and [pat] cases: they are needed for
the payload to be parsed as the right kind of node.
Consider now the extension node [[%expr 1 + 1]] in an expression context.
[Metaquot] will actually expand it into the following code:
{[
{
pexp_desc =
(Pexp_apply
({
pexp_desc = (Pexp_ident { txt = (Lident "+"); loc });
pexp_loc = loc;
pexp_attributes = []
},
[(Nolabel,
{
pexp_desc = (Pexp_constant (Pconst_integer ("1", None)));
pexp_loc = loc;
pexp_attributes = []
});
(Nolabel,
{
pexp_desc = (Pexp_constant (Pconst_integer ("1", None)));
pexp_loc = loc;
pexp_attributes = []
})]));
pexp_loc = loc;
pexp_attributes = []
}
]}
Looking at the example, you might notice two things:
- The AST types are used without a full path to the module.
- There is a free variable named [loc] and of type [Location.t] in the code.
So for this to compile, you need both to open [ppxlib] and to have a [loc : Location.t]
variable in scope.
The produced AST node value, and every other node within it, will be located in
this [loc]. You should therefore make sure that [loc] is the location you want for
your generated code when using [metaquot].
{2:antiquotations Anti-Quotations}
Using these extensions alone, you can only produce constant/static AST nodes.
[metaquot] has a solution for that: anti-quotation. You can use anti-quotation
to insert any expression representing an AST node. That way, you can include
dynamically generated nodes inside a [metaquot] expression extension point.
Consider the following example:
{[
let with_suffix_expr ~loc s =
let dynamic_node = Ast_builder.Default.estring ~loc s in
[%expr [%e dynamic_node] ^ "some_fixed_suffix"]
]}
The [with_suffix_expr] function will create an [expression] which represents the
concatenation of the [s] argument and the fixed suffix, i.e.,
[with_suffix_expr "some_dynamic_stem"] is equivalent to
[[%expr "some_dynamic_stem" ^ "some_fixed_suffix"]].
The syntax for anti-quotation depends on the type of the node you wish to insert
(which must also correspond to the context of the anti-quotation extension node):
- [e] is the extension point used to anti-quote values of type
{{!Ppxlib.Parsetree.expression}[expression]}:
{[let f some_expr_node = [%expr 1 + [%e some_expr_node]]]}
- [p] is the extension point used to anti-quote values of type
{{!Ppxlib.Parsetree.pattern}[pattern]}:
{[let f some_pat_node = [%pat? (1, [%p some_pat_node])]]}
- [t] is the extension point used to anti-quote values of type
{{!Ppxlib.Parsetree.core_type}[core_type]}:
{[let f some_core_type_node [%type: int -> [%t some_core_type_node]]]}
- [m] is the extension point used to anti-quote values of type
{{!Ppxlib.Parsetree.module_expr}[module_expr]}
or {{!Ppxlib.Parsetree.module_type}[module_type]}:
{[
let f some_module_expr_node = [%expr let module M = [%m some_module_expr_node] in M.x]
let f some_module_type_node = [%sigi: module M : [%m some_module_type_node]]
]}
- [i] is the extension point used to anti-quote values of type
{{!Ppxlib.Parsetree.structure_item}[structure_item]} or
{{!Ppxlib.Parsetree.signature_item}[signature_item]}. Note that the syntax for structure/signature item extension nodes uses two [%%]:
{[
let f some_structure_item_node =
[%str
let a = 1
[%%i some_structure_item_node]]
let f some_signature_item_node =
[%sig:
val a : int
[%%i some_signature_item_node]]
]}
If an anti-quote extension node is in the wrong context, it won't be
rewritten by {{!Ppxlib_metaquot}[Metaquot]}. For instance, in [[%expr match [] with [%e some_value] -> 1]]
the anti-quote extension node for expressions is put in a pattern context,
and it won't be rewritten.
On the contrary, you should use anti-quotes whose kind ([[%e ...]], [[%p ...]])
match the context. For example, you should write:
{@ocaml[
let let_generator pat type_ expr =
[%stri let [%p pat] : [%t type_] = [%e expr]] ;;
]}
Finally, remember that we are inserting values, so we never use patterns in the payloads of anti-quotations. Those will be used for {{!page-"matching-code".antiquotations}matching}.
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"writing-ppxs"}< Writing PPXs}{%html: </div><div>%}{{!"matching-code"}Destructing AST nodes >}{%html: </div></div>%}

346
unikernel/duniverse/ppxlib/doc/good-practices.mld Normal file
View file

@ -0,0 +1,346 @@
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"ast-traversal"}< Traversing the AST}{%html: </div><div>%}{{!"examples"}Examples >}{%html: </div></div>%}
{0 Good Practices}
{1:resp_loc Respecting Locations}
Correctly dealing with location is essential to correctly generate OCaml code.
They are necessary for error reporting by the compiler, but more generally for
Merlin's features to work, such as displaying occurrences and jumping to
definition. When called, the driver is called with the [-check] and
[-check-locations] flags, [ppxlib] makes it is a requirement that locations follow
some rules in order to accept the rewriting, as it will check that some
invariants are respected.
{2 The Invariants}
The invariants are as follows:
- AST nodes are requested to be well-nested WRT locations
- the locations of "sibling" AST nodes should not overlap
This is required for Merlin to behave properly.
Indeed, for almost any query directed at Merlin, it will need to inspect the
context around the user's cursor to give an answer that makes sense. And the
only input it has to do that is the cursor’s position in the buffer.
The handling of most queries starts by traversing the AST, using the
locations of nodes to select the right branch. (1) is necessary to avoid
discarding subtrees too early, (2) is used to avoid Merlin making arbitrary
choices (if you ask for the type under the cursor, and there seems to be two
things under the cursor, Merlin will need to pick one).
{2 Guidelines for Writing Well-Behaved PPXs}
It's obviously not always (indeed rarely) possible to mint new locations
when manipulating the AST.
The intended way to deal with locations is this:
- AST nodes that exist in the source should keep their original location
- new nodes should be given a "ghost" location (i.e.,
[{ some_loc with loc_ghost = true }]) to indicate that the node doesn't
exist in the sources.
In particular, {{!Ppxlib.Location.none}[Location.none]} is never meant to be
used by PPX authors, where some location is always available (for instance,
derivers and extenders at least know the locations of their relevant node).
Both the new check and Merlin will happily traverse the ghost nodes as if they
didn't exist. Note: this comes into play when deciding which nodes are
"siblings," for instance, if your AST is:
{v
A (B1(C, D),
B2(X, Y))
v}
but [B2] has a ghost location, then [B1], [X] and [Y] are considered
siblings.
Additionally, there is an attribute [\[@merlin.hide\]] that you can add on
nodes to tell Merlin (and the check) to ignore this node and all of its
children. Some helpers for this are provided in {{!Ppxlib.Merlin_helpers}[Merlin_helpers]}.
{1:handling_errors Handling Errors}
In order to give a nice user experience when using a PPX, it is necessary that
the resulting parsetree is as complete as possible. Most IDE tools, such as
Merlin, rely on the AST for their features, such as displaying type, jumping to
definition, or showing the list of errors.
In order to achieve this, errors that happen during rewriting should be handled
in a way that do not prevent a meaningful AST to be passed to Merlin.
There are mainly two ways to report errors when writing a PPX.
- By embedding special extensions nodes, called "error nodes", inside the
generated code.
- By raising a sepcific exception, letting the ppxlib driver {{!page-driver.exception_handling}handle the error}.
Let us emphasize that, while exceptions can be practical to quickly fail with an
error, the embedding mechanism has many advantages. For instance, embedding
allows to report multiple errors, and to output the part of the code that could
be generated successfully.
{2 Embedding the Errors in the AST}
It is better to always return a valid AST, as complete as possible, but with
"error extension nodes" at every place where successful code generation was
impossible. Error extension nodes are special extension nodes
[[%ocaml.error "error_message"]] that can be embedded into a valid AST and are interpreted later
as errors, e.g., by the compiler or Merlin. As all extension nodes, they can be
put {{:https://ocaml.org/manual/extensionnodes.html}at many places in the AST}
to replace structure items, expressions, or patterns, for example.
So whenever you're in doubt whether to throw an exception or if to embed the error as
an error extension node when writing a PPX rewriter,
embed the error is the way to go! And whenever you're in doubt about where
exactly to embed the error inside the AST, a good ground rule is: as deep in
the AST as possible.
For instance, suppose a rewriter is supposed to define a new record type, but
there is an error in one field’s type generation. In order to have
the most complete AST as output, the rewriter can still define the type and all
of its fields, putting an extension node in place of the type of the faulty
field:
{[
type long_record = {
field_1: int;
field_2: [%ocaml.error "field_2 could not be implemented due to foo"];
}
]}
[ppxlib] provides a function in its API to create error extension nodes:
{{!Ppxlib.Location.error_extensionf}[error_extensionf]}. This function creates
an extension node, which then must be transformed in the right kind of node
using functions such as
{{!Ppxlib.Ast_builder.Default.pexp_extension}[pexp_extension]}.
{2 A Documented Example}
Let us give an example. We will define a deriver on types records, which
constructs a default value from a given type. For instance, the derivation on
the type [type t = { x:int; y: float; z: string}] would yield [let default_t =
{x= 0; y= 0.; z= ""}]. This deriver has two limitations:
{ol
{- It does not work on other types than records,}
{- It only works for records containing fields of type [string], [int], or [float].}
}
The rewriter should warn the user about these limitations with a good error
reporting. Let’s first look at the second point. Here is the function mapping
the fields from the type definition to a default expression.
{[
let create_record ~loc fields =
let declaration_to_instantiation (ld : label_declaration) =
let loc = ld.pld_loc in
let { pld_type; pld_name; _ } = ld in
let e =
match pld_type with
| { ptyp_desc = Ptyp_constr ({ txt = Lident "string"; _ }, []); _ } ->
pexp_constant ~loc (Pconst_string ("", loc, None))
| { ptyp_desc = Ptyp_constr ({ txt = Lident "int"; _ }, []); _ } ->
pexp_constant ~loc (Pconst_integer ("0", None))
| { ptyp_desc = Ptyp_constr ({ txt = Lident "float"; _ }, []); _ } ->
pexp_constant ~loc (Pconst_float ("0.", None))
| _ ->
pexp_extension ~loc
@@ Location.error_extensionf ~loc
"Default value can only be derived for int, float, and string."
in
({ txt = Lident pld_name.txt; loc }, e)
in
let l = List.map fields ~f:declaration_to_instantiation in
pexp_record ~loc l None
]}
When the record definition contains several fields with types other than [int],
[float], or [string], several error nodes are added in the AST. Moreover, the
location of the error nodes corresponds to the field record's definition.
This allows tools such as Merlin to report all errors at once, at the right
location, resulting in a better workflow than having to recompile every time an
error is corrected to see the next one.
The first limitation is that the deriver cannot work on non-record types.
However, we decided here to derive a default value, even in the case of
non-record types, so that it does not appear as undefined in the remaining of
the file. This impossible value consists of an error extension node.
{[
let generate_impl ~ctxt (_rec_flag, type_declarations) =
let loc = Expansion_context.Deriver.derived_item_loc ctxt in
List.map type_declarations ~f:(fun (td : type_declaration) ->
let e, name =
match td with
| { ptype_kind = Ptype_record fields; ptype_name; ptype_loc; _ } ->
(create_record ~loc:ptype_loc fields, ptype_name)
| { ptype_name; ptype_loc; _ } ->
( pexp_extension ~loc
@@ Location.error_extensionf ~loc:ptype_loc
"Cannot derive accessors for non record type %s"
ptype_name.txt,
ptype_name )
in
[
pstr_value ~loc Nonrecursive
[
{
pvb_pat = ppat_var ~loc { txt = "default_" ^ name.txt; loc };
pvb_expr = e;
pvb_attributes = [];
pvb_loc = loc;
};
];
])
|> List.concat
]}
{1:quoting Quoting}
Quoting is part of producing
{{:https://en.wikipedia.org/wiki/Hygienic_macro}hygienic} code. But before
talking about the solution, let's introduce the problem.
Say you are writing an extension rewriter, which takes an expression as payload, and would replace all identifiers [id] in the expression with a similar expression, but with a printing debug:
{[
let x = 0 in
let y = 2 in
[%debug x + 1, y + 2 ]
]}
would generate the following code:
{[
let x = 0 in
let y = 2 in
let debug = Printf.printf "%s = %d; " in
(debug "x" x ; x) + 1,
(debug "y" y ; y) + 2
]}
When executed, the code would print [x = 0; y = 2; ]. So far, so good. However, suppose now that instead of [x], the variable is named [debug]. The following seemingly equivalent code:
{[
let debug = 0 in
let y = 2 in
[%debug debug + 1, y + 2 ]
]}
would generate:
{[
let debug = 0 in
let y = 2 in
let debug = Printf.printf "%s = %d; " in
(debug "debug" debug ; debug) + 1,
(debug "y" y ; y) + 2
]}
which does not even type-check! The problem is that the payload is expected to
be evaluated in some environment where [debug] has some value and type, but the
rewriting modifies this environment and shadows the [debug] name.
"Quoting" is a mechanism to prevent this problem from happenning. In [ppxlib], it
is done through the {{!Ppxlib.Expansion_helpers.Quoter}[Expansion_helpers.Quoter]} module in several steps:
- First, create a quoter using the {{!Ppxlib.Expansion_helpers.Quoter.create}[create]} function:
{[
# open Expansion_helper ;;
#s let quoter = Quoter.create () ;;
val quoter : Quoter.t = <abstr>
]}
- Then, use {{!Ppxlib.Expansion_helpers.Quoter.quote}[Expansion_helpers.Quoter.quote]} to quote all the expressions that are given from the user, might rely on a context, and that you want "intact."
{[
# let quoted_part = Quoter.quote quoter part_to_quote ;;
val quoted_payload : expression =
]}
- Finally, call {{!Ppxlib.Expansion_helpers.Quoter.sanitize}[Expansion_helpers.Quoter.sanitize]} on the whole expression (with quoted parts).
{[
# let result = Expansion_helpers.Quoter.sanitize ~quoter rewritten_expression ;;
val result : expression =
...
]}
If the [debug] rewriter had been written using this method, the quoting would
have ensured that the payload is evaluated in the same context as the
extension node!
Here is an example on how to write a [debug] rewriter (with the limitation that the payload should not contain variable binding, but the code was left simple to illustrate quoting):
{[
# let rewrite expr =
(* Create a quoter *)
let quoter = Quoter.create () in
(* An AST mapper to log and replace variables with quoted ones *)
let replace_var =
object
(* See the chapter on AST traverse *)
inherit Ast_traverse.map as super
(* in case of expression *)
method! expression expr =
match expr.pexp_desc with
(* in case of identifier (not "+") *)
| Pexp_ident { txt = Lident var_name; loc }
when not (String.equal "+" var_name) ->
(* quote the var *)
let quoted_var = Quoter.quote quoter expr in
let name = Ast_builder.Default.estring ~loc var_name in
(* and rewrite the expression *)
[%expr
debug [%e name] [%e quoted_var];
[%e quoted_var]]
(* otherwise, continue inside recursively *)
| _ -> super#expression expr
end
in
let quoted_rewrite = replace_var#expression expr in
let loc = expr.pexp_loc in
(* Sanitize the whole thing *)
Quoter.sanitize quoter
[%expr
let debug = Printf.printf "%s = %d; " in
[%e quoted_rewrite]] ;;
val rewrite : expression -> expression = <fun>
]}
With {!Ppxlib}'s current quoting mechanism, the code given in that example would look like:
{[
# Format.printf "%a\n" Pprintast.expression @@ rewrite [%expr debug + 1, y + 2] ;;
let rec __1 = y
and __0 = debug in
let debug = Printf.printf "%s = %d; " in
(((debug "debug" __0; __0) + 1), ((debug "y" __1; __1) + 2))
- : unit = ()
]}
{1 Testing Your PPX}
This section is not yet written. You can refer to {{:https://tarides.com/blog/2019-05-09-an-introduction-to-ocaml-ppx-ecosystem#testing-your-ppx}this blog post} (notice that that blog post was written before `dune` introduced its cram test feature), or contribute to the [ppxlib] documentation by opening a pull request in the {{:https://github.com/ocaml-ppx/ppxlib/}repository}.
{1 Migrate From Other Preprocessing Systems}
This section is not yet written. You can contribute to the [ppxlib] documentation by opening a pull request in the {{:https://github.com/ocaml-ppx/ppxlib/}repository}.
{1 Other good practices}
There are many good practices or other way to use [ppxlib] that are not mentioned in this manual. For instance, (in very short), you should always try to fully qualify variable names that are generated into the code via a PPX.
if you want to add a section to this "good practices" manual, you can contribute to the [ppxlib] documentation by opening a pull request in the {{:https://github.com/ocaml-ppx/ppxlib/}repository}.
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"ast-traversal"}< Traversing the AST}{%html: </div><div>%}{{!"examples"}Examples >}{%html: </div></div>%}

30
unikernel/duniverse/ppxlib/doc/index.mld Normal file
View file

@ -0,0 +1,30 @@
{0 [ppxlib]'s user manual}
{1 Overview}
This is the user manual and API for [ppxlib], the core of the PPX meta-programming
system for {{:https://ocaml.org/}OCaml} and its derivatives, such as
{{:https://reasonml.github.io/}Reason}. For a good introduction on PPXs, what
they are, and how to use them, see the
{{:https://ocaml.org/docs/metaprogramming}OCaml official guide} on PPXs. This
manual is mostly aimed at authors of PPX rewriters and contains everything one
should know in order to write PPX rewriters.
{1 Manual}
The manual consists of several sections. It can be read linearly, but you can also jump directly to your section of interest:
{ol
{li {{!page-"quick_intro"}An introduction to [ppxlib]}}
{li {{!page-"driver"}How [ppxlib] works internally}}
{li {{!page-"writing-ppxs"}Registering a transformation}}
{li {{!page-"generating-code"}Generating AST nodes}}
{li {{!page-"matching-code"}Destructing AST nodes}}
{li {{!page-"ast-traversal"}Traversing the AST}}
{li {{!page-"good-practices"}Good practices}}
{li {{!page-"examples"}Examples}}
}
{1 API}
The API exposes the following modules:
{!modules: ppxlib ppxlib_ast astlib ppxlib_metaquot Ppxlib_metaquot_lifters Ppxlib_print_diff Ppxlib_runner Ppxlib_runner_as_ppx Stdppx Ppxlib_traverse Ppxlib_traverse_builtins}

519
unikernel/duniverse/ppxlib/doc/matching-code.mld Normal file
View file

@ -0,0 +1,519 @@
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"generating-code"}< Generating AST nodes}{%html: </div><div>%}{{!"ast-traversal"}Traversing the AST >}{%html: </div></div>%}
{0 Destructing AST Nodes}
In the previous chapter, we have seen how to generate code. However, the
transformation function should depend on its input (the payload and maybe the
derived item), which we have to be able to inspect.
Once again, directly inspecting the {{!Ppxlib.Parsetree}[Parsetree]} value that
we get as input is not a good option because it is very big to manipulate and can
break at every new OCaml release. For instance, let's consider the case of
{{:https://github.com/janestreet/ppx_inline_test}[ppx_inline_test]}. We want to
recognize and extract the name and expression only from the form patterns:
{[
[%%test let "name" = expr]
]}
If we wrote a function accepting the payload of [[%%test]], and extracting the
name and expression from it, using normal pattern matching we would have:
{[
# let match_payload ~loc payload =
match payload with
| PStr
[
{
pstr_desc =
Pstr_value
( Nonrecursive,
[
{
pvb_pat =
{
ppat_desc =
Ppat_constant (Pconst_string (name, _, None));
_;
};
pvb_expr = expr;
_;
};
] );
_;
};
] ->
Ok (name, expr)
| _ -> Error (Location.Error.createf ~loc "Wrong pattern") ;;
val match_payload :
loc:location -> payload -> (string * expression, Location.Error.t) result =
]}
[ppxlib]'s solution to the verbosity and stability problem is to provide helpers
to {e match} the AST, in a very similar way to what it does for generating AST
nodes.
{1 The Different Options}
In this chapter, we will often mention the similarities between matching code
and generating code (from the {{!"generating-code"}previous chapter}). Indeed, the
options provided by [ppxlib] to match AST nodes mirror the ones for generating
nodes:
- {{!Ppxlib.Ast_pattern}[Ast_pattern]}, the {{!Ppxlib.Ast_builder}[Ast_builder]} sibling,
- {{!Ppxlib_metaquot}[Metaquot]} again.
{{!Ppxlib.Ast_pattern}[Ast_pattern]} is used in {{!Ppxlib.Extension.V3.declare}[Extension.V3.declare]}, so you will need it to write
extenders. {!Ppxlib_metaquot} is, as for generating nodes, more natural to use but also
restricted to some cases.
{1:ast_pattern_intro The [Ast_pattern] Module}
A match is a "structural destruction" of a value into multiple subvalues to
continue the computation. For instance, in the example above from the single
variable [payload], we structurally extract two variables: [name] and [expr].
Destruction is very similar to construction, but in reverse. Instead of using
several values to build a bigger one, we use one big value to define smaller
ones. As an illustration, note how in OCaml the following construction and
destruction are close:
{[
let big = { x ; y } (** Construction from [x] and [y] *)
let { x ; y } = big (** Destruction recovering [x] and [y] *)
]}
For the same reason, building AST nodes using {{!Ppxlib.Ast_builder}[Ast_builder]} and destructing AST
nodes using {{!Ppxlib.Ast_pattern}[Ast_pattern]} look very similar. The difference is that in the construction "leaf," {{!Ppxlib.Ast_builder}[Ast_builder]} uses actual values, while {{!Ppxlib.Ast_pattern}[Ast_pattern]} has
"wildcards" at the leafs.
Consider the example in the introduction matching [[%%test let "name" = expr]].
Building such an expression with {{!Ppxlib.Ast_builder}[Ast_builder]} could look like:
{[
# let build_payload_test ~loc name expr =
let (module B) = Ast_builder.make loc in
let open B in
Parsetree.PStr
(pstr_value Nonrecursive
(value_binding ~pat:(pstring name) ~expr :: [])
:: []) ;;
val build_payload_test :
loc:location -> string -> expression -> payload =
<abstr>
]}
Constructing a first-class pattern is almost as simple as replacing
[Ast_builder] with [Ast_pattern], as well as replacing the base values [name] and [expr] with a
capturing wildcard:
{[
# let destruct_payload_test () =
let open Ast_pattern in
pstr
(pstr_value nonrecursive
(value_binding ~pat:(pstring __) ~expr:__ ^:: nil)
^:: nil) ;;
val destruct_payload_test :
unit -> (payload, string -> expression -> 'a, 'a) Ast_pattern.t =
<abstr>
]}
Note that to facilitate viewing the similarity, we wrote [[v]] as [v :: []], and
we added a [unit] argument to avoid
{{:https://v2.ocaml.org/manual/polymorphism.html#ss:valuerestriction}value
restriction} to mess with the type (that we explained right in the next section).
{2 The Type for Patterns}
The {{!Ppxlib.Ast_pattern.t}[Ast_pattern.t]} type reflects the fact that a pattern-match or destruction
is taking a value, extracting other values from it, and using them to finally
output something. So, a value [v] of type [(matched, cont, res) Ast_pattern.t]
means that:
- The type of values matched by [v] is [matched]. For instance, [matched] could
be {{!Ppxlib.Parsetree.payload}[payload]}.
- The continuation (what to do with the extracted values) has type [cont]. The
values extracted from the destruction are passed as an argument to the
continuation, therefore [cont] includes information about them. For instance,
for a pattern that captures an [int] and a [string], [cont] could be
[int -> string -> structure]. The continuation is not part of [v]; it will
be given with the value to match.
- The result of the computation has type [res]. Note that this is additional information
than what we have in [cont]: {{!Ppxlib.Ast_pattern.map_result}[Ast_pattern.map_result]}
allows mapping the continuation result through a function! This allows users to add a
"construction" post-processing to the continuation. A value of type
[(pattern, int -> int, expression) Ast_pattern.t] would contain how to extract an integer from a [pattern] and how to map a modified [int] into an [expression].
In the case of the example above, [destruct_payload_test] has type:
{[
# destruct_payload_test ;;
val destruct_payload_test :
(payload, string -> expression -> 'a, 'a) Ast_pattern.t =
<abstr>
]}
as it destructs values
of type [pattern] extracts two values, respectively, of type [string] and
[expression], so the continuation has type [string -> expression -> 'a]. Then the
result type is ['a] since no mapping on the result is made. Now that the type of {{!Ppxlib.Ast_pattern.t}[Ast_pattern.t]} is explained, the type of
{{!Ppxlib.Ast_pattern.parse_res}[Ast_pattern.parse_res]}, the function for applying patterns, should make sense:
{@ocaml[
# Ast_pattern.parse_res ;;
val parse_res :
( 'matched, 'cont, 'res ) t ->
Location.t ->
?on_error:( unit -> 'res) ->
'matched ->
'cont ->
( 'res, Location.Error.t Stdppx.NonEmptyList.t ) result =
<fun>
]}
This function takes a pattern expecting values of type ['matched], continuations of
type ['cont] and output values of type [('res, _) result] (where the error case is when the ['matched] value does not have the expected structure).
The types of the function's other arguments correspond to this understanding: the argument of type ['matched] is
the value to match, the one of type ['cont] is the continuation, and the result
of applying the pattern to those two values is of type ['res]!
Composing construction and destruction yield the identity:
{@ocaml[
# let f name expr =
Ast_pattern.parse_res
(destruct_payload_test ()) Location.none
(build_payload_test ~loc name expr)
(fun name expr -> (name, expr)) ;;
val f :
string ->
expression ->
(string * expression, _) result = <fun>
# f "name" [%expr ()] ;;
Ok
("name",
{pexp_desc =
Pexp_construct
({txt = Lident "()";
...}...)...}...)
]}
While the {{!Ppxlib.Ast_pattern.parse_res}[Ast_pattern.parse_res]} function is useful to match an AST node, you
will also need the {{!Ppxlib.Ast_pattern.t}[Ast_pattern.t]} value in other contexts. For instance, it is
used when declaring extenders with {{!Ppxlib.Extension.declare}[Extension.declare]} to tell how to extract
arguments from the payload to give them to the extender, or when parsing with {{!Ppxlib.Deriving.Args.arg}deriving arguments}.
{2 Building Patterns}
Now that we know what these patterns represent and how to use them, and have seen an
example in the {{!ast_pattern_intro}introduction} on {{!Ppxlib.Ast_pattern}[Ast_pattern]}, the
combinators in the {{!Ppxlib.Ast_pattern}API} should be much more easily
understandable. So, for a comprehensive list of the different values in the
module, the reader should directly refer to the API. In this guide; however, we
explain in more detail a few important values with examples.
{b The wildcard pattern [| x -> ]}. The simplest way to extract a value from something
is just to return it! In {{!Ppxlib.Ast_pattern}[Ast_pattern]}, it corresponds to the value
{{!Ppxlib.Ast_pattern.__}[__]} (of type [('a, 'a -> 'b, 'b)]), which extract the
value it's given: {{!Ppxlib.Ast_pattern.parse_res}matching} a value [v]
with this pattern and a continuation [k] would simply call [k v].
This pattern is useful in combination with other combinators.
{b The wildcard-dropping pattern [| _ -> ]}. Despite their name ressemblance,
{{!Ppxlib.Ast_pattern.__}[__]} is very different from the OCaml pattern-match
wildcard [_], which accepts everything but {e ignores} its input. In {{!Ppxlib.Ast_pattern}[Ast_pattern]},
the wildcard-dropping pattern is {{!Ppxlib.Ast_pattern.drop}[drop]}. Again, it
is useful in conjunction with other combinators, where one needs
to accept all input in some places, but the value is not relevant.
{b The [| p as name -> ] combinator}. The combinator {{!Ppxlib.Ast_pattern.as__}[as__]}
allows passing a node to the continuation while still extracting values from
this node. For instance, [as__ (some __)] corresponds to the OCaml pattern-match
[ Some n2 as n1], where the continuation is called with [k n1 n2].
{b The [| (p1 | p2) -> ] combinator}. The combinator {{!Ppxlib.Ast_pattern.alt}[alt]}
combines two patterns with the same type for extracted values into one pattern
by first trying to apply the first, and if it fails, by applying the second one.
For instance, [alt (pair (some __) drop) (pair drop (some __))] corresponds to
the OCaml pattern [(Some a, _) | (_, Some b)].
{b The constant patterns [| "constant" -> ]}. Using {{!Ppxlib.Ast_pattern.cst}[Ast_pattern.cst]} it is
possible to create patterns matching only fixed values, such as the ["constant"]
string. No values are extracted from this matching. The functions for creating
such values are {{!Ppxlib.Ast_pattern.int}[Ast_pattern.int]}, {{!Ppxlib.Ast_pattern.string}[Ast_pattern.string]}, {{!Ppxlib.Ast_pattern.bool}[Ast_pattern.bool]}, ...
{b The common deconstructors}. Many usual common constructors have
"deconstructors" in {{!Ppxlib.Ast_pattern}[Ast_pattern]}. For instance:
- [some __] corresponds to [Some a],
- [__ ^:: drop ^:: nil] correspnds to [a :: _ :: []],
- [pair __ __] (or equivalently [__ ** __]) corresponds to [(a,b)], etc.
{b The Parsetree deconstructors}. All constructors from {{!Ppxlib.Ast_builder}[Ast_builder]} have a
"deconstructor" in {{!Ppxlib.Ast_pattern}[Ast_pattern]} with the same name. For instance, since
{{!Ppxlib.Ast_builder}[Ast_builder]} has a constructor {{!Ppxlib.Ast_builder.Default.pstr_value}[pstr_value]} to build a structure
item from a [rec_flag] and a [value_binding] list. {{!Ppxlib.Ast_pattern}[Ast_pattern]} has an equally
named {{!Ppxlib.Ast_pattern.pstr_value}[pstr_value]} which, given ways to destruct rec flags and
[value_binding] lists, creates a destructor for structure items.
{b The continuation modifiers}. Many {{!Ppxlib.Ast_pattern}[Ast_pattern]} values allow modifying the
continuation. It can be it a map on the continuation itself, the argument to the
continuation, or the result of the continuation. So, {{!Ppxlib.Ast_pattern.map}[Ast_pattern.map]} transforms the
continuation itself, e.g., [map ~f:Fun.flip] will switch the arguments of
the function. {{!Ppxlib.Ast_pattern.map1}[map<i>]} modifies the arguments to a
continuation of arity [i]: [map2 ~f:combine] is equivalent to
[map ~f:(fun k -> (fun x y -> k (combine x y)))]. Finally, {{!Ppxlib.Ast_pattern.map_result}[Ast_pattern.map_result]} modifies
the continuation's result, and [map_result ~f:ignore] would ignore the continuation's result.
{b Common patterns} Some patterns are sufficiently common that, although they can be built from smaller bricks, they are already defined in {{!Ppxlib.Ast_pattern}[Ast_pattern]}. For instance, matching a single expression in a payload is given as {{!Ppxlib.Ast_pattern.single_expr_payload}[Ast_pattern.single_expr_payload]}.
{2:pattern_examples Useful patterns and examples}
Below, is a list of patterns that are commonly needed when using {{!Ppxlib.Ast_pattern}[Ast_pattern]}:
{@ocaml[
open Ast_pattern
]}
- A pattern to extract an expression from an extension point payload:
{@ocaml[
# let extractor () = single_expr_payload __ ;
val extractor : unit -> (payload, expression -> 'a, 'a) t = <fun>
]}
- A pattern to extract a string from an extension point payload:
{@ocaml[
# let extractor () = single_expr_payload (estring __) ;
val extractor : unit -> (payload, string -> 'a, 'a) t = <fun>
]}
- A pattern to extract a pair [int * float] from an extension point payload:
{@ocaml[
# let extractor () = single_expr_payload (pexp_tuple (eint __ ^:: efloat __ ^:: nil)) ;;
val extractor : unit -> (payload, int -> string -> 'a, 'a) t = <fun>
]}
- A pattern to extract a list of integers from an extension point payload, given
as a tuple (of unfixed length):
{@ocaml[
# let extractor () = single_expr_payload (pexp_tuple (many (eint __))) ;;
val extractor : unit -> (payload, int -> string -> 'a, 'a) t = <fun>
]}
- A pattern to extract a list of integers from an extension point payload, given
as a list:
{@ocaml[
# let extractor () = single_expr_payload (elist (eint __)) ;;
val extractor : unit -> (payload, int list -> 'a, 'a) t = <fun>
]}
- A pattern to extract the [pattern] and the [expression] in a let-binding, from a structure item:
{@ocaml[
# let extractor_in_let () = pstr_value drop ((value_binding ~pat:__ ~expr:__) ^:: nil);;
val extractor_in_let : unit -> (structure_item, pattern -> expression -> 'a, 'a) t =
<fun>
]}
- A pattern to extract the [pattern] and the [expression] in a let-binding, from an extension point payload:
{@ocaml[
# let extractor () = pstr @@ extractor_in_let ^:: nil;;
val extractor : unit -> (payload, pattern -> expression -> 'a, 'a) t = <fun>
]}
- A pattern to extract a core type, from an extension point payload (with a comma in the extension node, such as [[%ext_name: core_type]]):
{@ocaml[
# let extractor () = ptyp __
val extractor : unit -> (payload, core_type -> 'a, 'a) t = <fun>
]}
- A pattern to extract a string from an expression, either from an identifier or from a string. That is, it will extract the string ["foo"] from both the AST nodes [foo] and ["foo"].
{@ocaml[
# let extractor () = alt (pexp_ident (lident __)) (estring __) ;;
val extractor : unit -> (expression, string -> 'a, 'a) t = <fun>
]}
- A pattern to extract a sequence of two idents, as strings (will extract ["foo"], ["bar"] from [[%ext_name foo bar]]):
{@ocaml[
let extractor () =
single_expr_payload @@
pexp_apply
(pexp_ident (lident __))
((no_label (pexp_ident (lident __))) ^:: nil) ;;
val extractor : unit -> (payload, string -> string -> 'a, 'a) t = <fun>]}
{1:metaquot [Metaquot]}
{2 [Metaquot] for Patterns}
Recall that [ppxlib] provides a rewriter to generate code explained in
{{!page-"generating-code".metaquot}the corresponding chapter}. The same PPX can
also generate patterns when the extension nodes are used patterns: for
instance, in what follows, the extension node will be replaced by a value of {{!Ppxlib.Parsetree.expression}[expression]} type:
{[
let f = [%expr 1 + 1]
]}
While in the following, it would be replaced by a pattern matching on values of {{!Ppxlib.Parsetree.expression}[expression]} type:
{[
let f x = match x with
| [%expr 1 + 1] -> ...
| _ -> ...
]}
The produced pattern matches regardless of location and attributes. For
the previous example, it will produce the following pattern:
{[
{
pexp_desc =
(Pexp_apply
({
pexp_desc = (Pexp_ident { txt = (Lident "+"); loc = _ });
pexp_loc = _;
pexp_attributes = _
},
[(Nolabel,
{
pexp_desc = (Pexp_constant (Pconst_integer ("1", None)));
pexp_loc = _;
pexp_attributes = _
});
(Nolabel,
{
pexp_desc = (Pexp_constant (Pconst_integer ("1", None)));
pexp_loc = _;
pexp_attributes = _
})]));
pexp_loc = _;
pexp_attributes = _
}
]}
While being less general than {{!Ppxlib.Ast_pattern}[Ast_pattern]}, this allows users to write
patterns in a more natural way. Due to the OCaml AST, {{!Ppxlib.Parsetree.payload}payloads} can only
take the form of a {{!Ppxlib.Parsetree.structure}[structure]}, a {{!Ppxlib.Parsetree.signature}[signature]}, a {{!Ppxlib.Parsetree.core_type}[core type]}, or a {{!Ppxlib.Parsetree.pattern}[pattern]}. We might
want to generate pattern matching for other kinds of nodes, such as expressions or
structure item. The same extension nodes that [Metaquot] provides
for building can be used for matching:
- The [expr] extension node to match on {{!Ppxlib.Parsetree.expression}[expressions]}:
{[match expr with [%expr 1 + 1] -> ...]}
- The [pat] extension node to match on {{!Ppxlib.Parsetree.pattern}[patterns]}:
{[match pattern with [%pat? ("", _)] -> ...]}
- The [type] extension node to match on for {{!Ppxlib.Parsetree.core_type}[core types]}:
{[match typ with [%type: int -> string] -> ...]}
- The [stri] and [sigi] extension nodes to match on {{!Ppxlib.Parsetree.structure_item}[structure_item]} and {{!Ppxlib.Parsetree.signature_item}[signature_item]}:
{[match stri with [%stri let a = 1] -> ...
match sigi with [%sigi: val a : int] -> ...]}
- The [str] and [sig] extension nodes to match on
{{!Ppxlib.Parsetree.structure}[structure]}
and {{!Ppxlib.Parsetree.signature}[signature]}.
{[
let _ =
match str with
| [%str
let a = 1
let b = 2.1] ->
()
let _ =
match sigi with
| [%sigi:
val a : int
val b : float] ->
()
]}
{2:antiquotations Anti-Quotations}
{{!page-"generating-code".antiquotations}Similarly} to the [expression] context, these extension nodes have a limitation: when using these extensions alone, you can't
bind variables. [Metaquot] also solves
this problem using anti-quotation.
In the [pattern] context, anti-quotation is not used to insert values but to insert
patterns. That way you can include a wildcard or variable-binding pattern.
Consider the following example, which matches expression nodes corresponding to
the sum of three expressions: starting with the constant 1, followed by anything,
followed by anything bound to the [third] variable, which has type
[expression]:
{[
match some_expr_node with
| [%expr 1 + [%e? _] + [%e? third]] -> do_something_with third
]}
The syntax for anti-quotation depends on the type of the node you wish to insert
(which must also correspond to the context of the anti-quotation extension node):
- The extension point [e] is used to anti-quote values of type
{{!Ppxlib.Parsetree.expression}[expression]}:
{[match e with [%expr 1 + [%e? some_expr_pattern]] -> ...]}
- The extension point [p] is used to anti-quote values of type
{{!Ppxlib.Parsetree.pattern}[pattern]}:
{[match pat with [%stri let [%p? x] = [%e? y]] -> do_something_with x y]}
- The extension point [t] is used to anti-quote values of type
{{!Ppxlib.Parsetree.core_type}[core_type]}:
{[match t with [%type: int -> [%t? _]] -> ...]}
- The extension point [m] is used to anti-quote values of type
{{!Ppxlib.Parsetree.module_expr}[module_expr]}
or {{!Ppxlib.Parsetree.module_type}[module_type]}:
{[
let [%expr
let module M = [%m? extracted_m] in
M.x] =
some_expr
in
do_something_with extracted_m
let _ = fun [%sigi: module M : [%m? input]] -> do_something_with input
]}
- The extension point [i] is used to anti-quote values of type
{{!Ppxlib.Parsetree.structure_item}[structure_item]} or
{{!Ppxlib.Parsetree.signature_item}[signature_item]}:
{[
let [%str
let a = 1
[%%i? stri2]] =
e
in
do_something_with stri2
;;
let [%sig:
val a : int
[%%i? sigi2]] =
s
in
do_something_with sigi2
]}
Remember, since we are inserting patterns (and not expressions), we always use
patterns as payload, as in [[%e? x]].
If an anti-quote extension node is in the wrong context, it won't be rewritten
by [Metaquot]. For instance, in [fun [%expr 1 + [%p? x]] -> x] the
anti-quote extension node for the expression is put in a pattern context, and
it won't be rewritten.
On the contrary, you should use anti-quotes whose kind ([[%e ...]], [[%p ...]])
match the context. For example, you should write:
{@ocaml[
fun [%stri let ([%p pat] : [%t type_]) = [%e expr]] ->
do_something_with pat type_ expr
]}
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"generating-code"}< Generating AST nodes}{%html: </div><div>%}{{!"ast-traversal"}Traversing the AST >}{%html: </div></div>%}

84
unikernel/duniverse/ppxlib/doc/quick_intro.mld Normal file
View file

@ -0,0 +1,84 @@
{%html: <div style="display: flex; justify-content:space-between"><div>%}{%html: </div><div>%}{{!driver}The Driver >}{%html: </div></div>%}
{0 Introduction}
This guide is intended at authors or future authors of PPX rewriters. If you
don't know what a PPX is, or if you are looking for a guide intended at PPX
users, read the {{:https://ocaml.org/docs/metaprogramming}OCaml
official guide on meta-programming} first, although the beginning of this guide
may be of interest to everyone.
{1 Preprocessing in OCaml}
OCaml doesn't have a macro system, that is, there is no official part of the
OCaml language that will be executed at compile time in order to generate or
alter the source code. However, OCaml does have an official part of its syntax dedicated to this: {{:https://ocaml.org/manual/extensionnodes.html}extension nodes} and {{:https://ocaml.org/manual/attributes.html}attributes}; both of them introduced in OCaml 4.02. The preprocessing itself, though, is left to external programs,
written by the community and specialised for their own tasks. However, without a
unification framework, the following issues arise:
- Ambiguity when using several preprocessors due to lacking clear composition semantics
- Duplication of code and efforts on the different preprocessors
- Performance loss with many phases (parsing, pretty-printing, etc.) being
executed multiple times
- Lack of cross-compiler compatibility
- Incompatibility among the different preprocessor rather than one homogeneous preprocessor ecosystem
{1 [ppxlib]}
The goal of [ppxlib] is to solve these problems, by providing a unifying
framework for writing preprocessors. It sits in between the OCaml compiler and
toolchain, and the PPX authors provide an API for them. One could sum up the
[ppxlib] features as:
- It deals with all boilerplate, such as parsing the input, outputting the rewritten
output, generating an executable, etc.
- It generates a single executable for multiple transformations and defines clear composition semantics for local transformations
- It integrates well with Dune and Merlin
- It provides a more stable API than the compiler for manipulating the AST
- A single PPX codebase usually works on several OCaml versions of the AST
- It defines restricted rewriters whose semantic ensure better confidence for the
user and better compositional semantics
- It provides many helpers to pattern-match and generate AST nodes, as well as traverse the
AST.
{1 This Guide}
This guide is separated into several parts.
First, we focus on the {{!driver}driver} that performs the AST transformations:
how it is generated, executed, and more importantly, what it does exactly, from
migrating the AST to the different rewriting phases it goes through.
After that, we explain {{!"writing-ppxs"}the different kinds} of transformations
that [ppxlib] supports, and how to register them to the driver. This section only
describes the transformations and their properties, not how to actually
manipulate the AST.
The part where we discuss how to manipulate the AST is split in three pages:
{{!"generating-code"}generating AST nodes} to generate OCaml code,
{{!"matching-code"}destructing AST nodes} to extract information and act
differently depending on what is extracted, and {{!"ast-traversal"}traversing
the AST} to use [fold], [iter], and [map] on the AST. This code-manipulation part
explains using the modules {{!Ppxlib.Ast_builder}[Ast_builder]}, {{!Ppxlib.Ast_pattern}[Ast_pattern]}, and the [ppxlib]'s
PPX {{!Ppxlib_metaquot}[Metaquot]}.
We finally discuss several {{!"good-practices"}good practices}, such as how to
properly report errors, how to test your PPX, or how to migrate from other PPX
libraries, such as [OMP] and [ppx_deriving].
We end by including some {{!"examples"}examples}, which you can also find in the
[examples] {{:https://github.com/ocaml-ppx/ppxlib/tree/main/examples}folder} of
[ppxlib]'s repository.
{1 PPXs and [ppxlib] History}
The preprocessing history of OCaml started long before [ppxlib]. However, this
section is not yet written. You can find more information in these resources:
{{:https://lists.ocaml.org/pipermail/wg-camlp4/2013-January/000000.html}1}
{{:https://caml.inria.fr/pub/docs/manual-camlp4/manual002.html}2}
{{:https://camlp5.github.io/}3}
{{:https://discuss.ocaml.org/t/an-update-on-the-state-of-the-ppx-ecosystem-and-ppxlib-s-transition/8200}5}.
You can also contribute to the [ppxlib] documentation by opening a pull request in
the {{:https://github.com/ocaml-ppx/ppxlib/}repository}.
{%html: <div style="display: flex; justify-content:space-between"><div>%}{%html: </div><div>%}{{!driver}The Driver >}{%html: </div></div>%}

628
unikernel/duniverse/ppxlib/doc/writing-ppxs.mld Normal file
View file

@ -0,0 +1,628 @@
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"driver"}< The Driver}{%html: </div><div>%}{{!"generating-code"}Generating AST nodes >}{%html: </div></div>%}
{0 Writing a Transformation}
This chapter covers the [ppxlib] procedure basics to define and register a
transformation, be it a global or a context-free transformation.
For the actual manipulation and generation of code, [ppxlib] provides many helpers
that are listed in {!generatingcode}.
{1 Defining a Transformation}
For [ppxlib], a transformation is a description of a way to modify a given AST
into another one. A transformation can be:
- A context-free transformation, which only acts on a portion of the AST. In the [ppxlib] framework, those transformations
are represented by values of type {{!Ppxlib.Context_free.Rule.t}[Context_free.Rule.t]} and are executed in the {{!driver."context-free-phase"}context-free phase}. This is the strongly recommended kind of transformation due to its {{!driver.advantages}important advantages}, such as good performance, well-defined composition semantics, and the safety and trustability that comes with well-isolated and strictly local modifications.
- A global transformation, which takes the simple form of a function of type
[structure -> structure] or [signature -> signature], that can sometimes take
extra information as additional arguments. Such a transformation is applied in
the {{!driver."global-transfo-phase"}global transformation phase}, unless it
has a good reason to have been registered in another phase. While global transformations are a flexible and powerful tool in the OCaml ecosystem, they come with many {{!global_transformation}drawbacks} and should only be used when really necessary.
In order to register a transformation to the [ppxlib] driver, one should use the
{{!Ppxlib.Driver.V2.register_transformation}[Driver.V2.register_transformation]}. This function is used to register all
rewriter types in every different phase, except derivers, which are abstracted
away in {{!Ppxlib.Deriving}[Deriving]}.
{1 Context-Free Transformation}
In [ppxlib], the type for context-free transformation is
{{!Ppxlib.Context_free.Rule.t}[Context_free.Rule.t]}. Rules will be applied during the AST's top-down traverse
of the context-free pass. A rule contains the information about
when it should be applied in the traversal, as well as the transformation to
apply.
Currently, rules can only be defined to apply in five different contexts:
- on extensions points, such as [\[%ext_point payload\]]
- on some structure or signature items with a deriving attribute, such as
[type t = Nil \[@@deriving show\]],
- on AST nodes with attributes, such as [let x = 42 [@@attr]],
- on
{{:https://v2.ocaml.org/manual/extensionsyntax.html#ss:extension-literals}
litterals with modifiers}, such as [41g] or [43.2x],
- on function application or identifiers, such as [meta_function "99"] and [meta_constant].
In order to define rules on extensions points, we will use the {{!Ppxlib.Extension}[Extension]}
module. In order to define deriving rules, we will use the {{!Ppxlib.Deriving}[Deriving]}
module. For the three other rules, we will directly use the
{{!Ppxlib.Context_free.Rule}[Context_free.Rule]} module.
{2 Extenders}
An {{!driver.def_extenders}extender} is characterised by several things:
{ul
{li The situation that triggers the rewriting, which consists of two things:
{ul
{li The extension points' name on which it is triggered. For instance,
an extender triggered on [[%name]] would not be triggered on [[%other_name]]}
{li The AST context on which it applies. Indeed, extension points can be used in
many different places: expression, pattern, core type, etc., and the extender
should be restricted to one context, as it produces code of a single type. So,
an extender triggered on expressions could be triggered on [let x = [%name]]
but not on [let [%name] = expr].}}}
{li The actual rewriting of the extension node:
{ul
{li A function, called "expander", taking arguments and outputting the generated AST}
{li How to extract from the payload the arguments to pass to the expander}}
}}
{3:ext_context The Extender Context}
The context is a value of type {{!Ppxlib.Extension.Context.t}[Extension.Context.t]}. For instance, to
define an extender for expression-extension points, the correct context is
{{!Ppxlib.Extension.Context.expression}[Extension.Context.expression]}. Consult the
{{!Ppxlib.Extension.Context}[Extension.Context]} module's API for the list of all contexts!
{@ocaml[
# let context = Extension.Context.expression;;
val context : expression Extension.Context.t =
Ppxlib.Extension.Context.Expression
]}
{3 The Extender Name}
The extension point name on which it applies is simply a string.
{@ocaml[
# let extender_name = "add_suffix" ;;
val extender_name : string = "add_suffix"
]}
See below for examples on when the above name and context will trigger rewriting:
{@ocaml[
(* will trigger rewriting: *)
let _ = [%add_suffix "payload"]
(* won't trigger rewriting: *)
let _ = [%other_name "payload"] (* wrong name *)
let _ = match () with [%add_suffix "payload"] -> () (* wrong context *)
]}
{3 The Payload Extraction}
An extension node contains a {{!Ppxlib.Parsetree.payload}[payload]}, which will be passed to the transformation function. However, while this payload contains all information, it is not always structured the best way for the transformation function. For instance, in [[%add_suffix "payload"]], the string ["payload"] is encoded as a structure item consisting of an expression’s evaluation, a constant that is a string.
[ppxlib] allows separating the transformation function from the extraction of the payload’s relevant information. As explained in depth in the {{!"matching-code"}Destructing AST nodes} chapter, this extraction is done by destructing the payload’s structure (which is therefore restricted: [[%add_suffix 12]] would be refused by the rewriter of the example below). The extraction is defined by a value of type
{{!Ppxlib.Ast_pattern.t}[Ast_pattern.t]}. The {{!Ppxlib.Ast_pattern}[Ast_pattern]} module provides some kind of pattern-matching on AST nodes: a way to structurally extract values from an AST
node in order to generate a value of another kind.
For instance, a value of type
[(payload, int -> float -> expression, expression) Ast_pattern.t] means that it
defines a way to extract an [int] and a [float] from a {{!Ppxlib.Parsetree.payload}[payload]},
which should be then combined to define a value of type {{!Ppxlib.Parsetree.expression}[expression]}.
In our case, the matched value will always be a {{!Ppxlib.Parsetree.payload}[payload]}, as that's the type for extension points' payloads. The type of the
produced node will have to match the {{!ext_context}type of extension node we rewrite}, {{!Ppxlib.Parsetree.expression}[expression]} in our example.
{@ocaml[
# let extracter () = Ast_pattern.(single_expr_payload (estring __)) ;;
val extracter : unit -> (payload, string -> 'a, 'a) Ast_pattern.t = <fun>
]}
The above pattern extracts a string inside an extension node pattern. It will extract ["string"] in the the extension node [[%ext_name "string"]] and will refuse [[%ext_name 1+1]]. For other ready-to-use examples of patterns, refer to the {{!"matching-code".pattern_examples}example} section. For more in-depth explanation on the types and functions used above, see the {{!"matching-code"}Destructing AST nodes} chapter and the {{!Ppxlib.Ast_pattern}[Ast_pattern] API} .
The unit argument in [extractor] is not important. It is added so that {{:https://v2.ocaml.org/manual/polymorphism.html#ss:valuerestriction}value restriction} does not add noise to the type variables.
{3 The Expand Function}
The expander is the function that takes the values extracted from the
payload and produces the value that replaces the extension node.
Building and inspecting AST nodes can be painful due to how
{{!Ppxlib.Parsetree}large} the AST type is. [ppxlib] provides several helper
modules to ease this generation, such as {{!Ppxlib.Ast_builder}[Ast_builder]},
{!Ppxlib_metaquot}, {{!Ppxlib.Ast_pattern}[Ast_pattern]}, and {{!Ppxlib.Ast_traverse}[Ast_traverse]}, which are
explained in their own chapters: {{!"generating-code"}Generating AST nodes},
{{!"matching-code"}Destructing AST nodes} and {{!"ast-traversal"}Traversing AST nodes}.
In the example below, you can ignore the body of the function until reading
those chapters.
{@ocaml[
# let expander ~ctxt s =
let loc = Expansion_context.Extension.extension_point_loc ctxt in
Ast_builder.Default.(estring ~loc (s ^ "_suffixed")) ;;
val expander : ctxt:Expansion_context.Extension.t -> string -> expression =
<fun>
]}
The expander takes [ctxt] as a named argument that is ignored here. This
argument corresponds to additional information, such as the location of the
extension node. More precisely, it is of type
{{!Ppxlib.Expansion_context.Extension.t}[Expansion_context.Extension.t]} and
includes:
- The location of the extension node
- The tool that called the rewriting ([merlin], [ocamlc], [ocaml],
[ocamlopt], etc.)
- The name of the input file given to the driver (see
{{!Ppxlib.Expansion_context.Base.input_name}[Expansion_context.Base.input_name]})
- The [code_path] (see {{!Ppxlib.Expansion_context.Base.input_name}[Expansion_context.Base.input_name]} and
{{!Ppxlib.Code_path}[Code_path]})
{3 Declaring an Extender}
When we have defined the four prerequisites, we are able to combine all of them to define an
extender using the {{!Ppxlib.Extension.V3.declare}[Extension.V3.declare]} function.
{[
# V3.declare ;;
string ->
'context Context.t ->
(payload, 'a, 'context) Ast_pattern.t ->
(ctxt:Expansion_context.Extension.t -> 'a) ->
t
]}
Note that the type is consistent: the context on which the expander is
applied and the value produced by the expander need to be equal (indeed,
['a] must be of the form ['extacted_1 -> 'extracted_2 -> ... -> 'context]
with the constraints given by {{!Ppxlib.Ast_pattern}[Ast_pattern]}).
We are thus able to create the extender given by the previous examples:
{[
# let my_extender = Extension.V3.declare extender_name context (extracter()) expander ;;
val my_extender : Extension.t = <abstr>
]}
Note that we use the [V3] version of the [declare] function, which passes the
expansion context to the expander. Previous versions were kept for
retro-compatibility.
We can finally turn the extender into a rule (using
{{!Ppxlib.Context_free.Rule.extension}[Context_free.Rule.extension]}) and register it to the driver:
{[
# let extender_rule = Context_free.Rule.extension my_extender ;;
val extender_rule : Context_free.Rule.t = <abstr>
# Driver.register_transformation ~rules:[extender_rule] "name_only_for_debug_purpose" ;;
- : unit = ()
]}
Now, the following:
{@ocaml[
let () = print_endline [%add_suffix "helloworld"]
]}
would be rewritten by the PPX in:
{@ocaml[
let () = print_endline "helloworld_suffixed"
]}
{2 Derivers}
A {{!driver.def_derivers}deriver} is characterised by several things:
- The way to parse arguments passed through the attribute payload
- The set of other derivers that need to run before it is applied
- The actual generator function
Contrary to extenders, the registration of the deriver as a
{{!Ppxlib.Context_free.Rule.t}[Context_free.Rule.t]} is not made by the user via
{{!Ppxlib.Driver.register_transformation}[Driver.register_transformation]}, but
rather by {{!Ppxlib.Deriving.add}[Deriving.add]}.
{4 Derivers Arguments}
In [ppxlib], a deriver is applied by adding an attribute containing the derivers' names
to apply:
{[
type tree = Leaf | Node of tree * tree [@@deriving show, yojson]
]}
However, it is also possible to pass arguments to the derivers, either through a
record or through labelled arguments:
{[
type tree = Leaf | Node of tree * tree [@@deriving my_deriver ~flag ~option1:52]
]}
or
{[
type tree = Leaf | Node of tree * tree [@@deriving my_deriver { flag; option1=52 }]
]}
The [flag] argument is a flag, and it can only be present or absent but not take a
value. The [option1] argument is a regular argument, so it is also optional but can
take a value.
In [ppxlib], arguments have the type {{!Ppxlib.Deriving.Args.t}[Deriving.Args.t]}. Similarly to the
{{!Ppxlib.Ast_pattern.t}[Ast_pattern.t]} type, a value of type [(int -> string -> structure, structure) Args.t]
means that it provides a way to extract an integer from the argument and
a string from the options, later combined to create a structure.
The way to define a {{!Ppxlib.Deriving.Args.t}[Deriving.Args.t]} value is to start with the value describing an
empty set of arguments, {{!Ppxlib.Deriving.Args.empty}[Deriving.Args.empty]}. Then add the arguments one by one, using
the combinator {{!Ppxlib.Deriving.Args.(+>)}[Deriving.Args.(+>)]}. Each argument is created using either {{!Ppxlib.Deriving.Args.arg}[Deriving.Args.arg]}
for optional arguments (with value extracted using {{!Ppxlib.Ast_pattern}[Ast_pattern]}) or
{{!Ppxlib.Deriving.Args.flag}[Deriving.Args.flag]} for optional arguments without values.
{@ocaml[
# let args () = Deriving.Args.(empty +> arg "option1" (eint __) +> flag "flag") ;;
val args : (int option -> bool -> 'a, 'a) Deriving.Args.t = <abstr>
]}
{4 Derivers Dependency}
[ppxlib] allows declaring that a deriver depends on the previous application of
another deriver. This is expressed simply as a list of derivers. For instance,
the {{:https://github.com/janestreet/ppx_csv_conv}csv} deriver depends on the
{{:https://github.com/janestreet/ppx_fields_conv}fields} deriver to run
first.
{@ocaml[
# let deps = [] ;;
val deps : 'a list = []
]}
In this example, we do not include any dependency.
{3 Generator Function}
Similarly to an extender's [expand] function, the function generating new
code in derivers also takes a context and the arguments extracted from the
attribute payload. Here again, the body of the example function can be
safely ignored ,as it relies on {{!"generating-code"}later chapters}.
{@ocaml[
# let generate_impl ~ctxt _ast option1 flag =
let return s = (* See "Generating code" chapter *)
let loc = Expansion_context.Deriver.derived_item_loc ctxt in
[ Ast_builder.Default.(pstr_eval ~loc (estring ~loc s) []) ]
in
if flag then return "flag is on"
else
match option1 with
| Some i -> return (Printf.sprintf "option is %d" i)
| None -> return "flag and option are not set" ;;
val generate_impl :
ctxt:Expansion_context.Deriver.t ->
'a -> int option -> bool -> structure_item list = <fun>
]}
Similarly to extenders, there is an additional (ignored
in the example) argument to the function: the context. This time, the context
is of type {{!Ppxlib.Expansion_context.Deriver.t}[Expansion_context.Deriver.t]} and includes:
- The location of the derived item
- Whether the code generation will be inlined (see {!"inlining-transformations"})
- The tool that called the rewriting ([merlin], [ocamlc], [ocaml],
[ocamlopt], etc.),
- The name of the input file given to the driver (see
{{!Ppxlib.Expansion_context.Base.input_name}[Expansion_context.Base.input_name]})
- The [code_path] (see {{!Ppxlib.Expansion_context.Base.input_name}[Expansion_context.Base.input_name]} and
{{!Ppxlib.Code_path}[Code_path]}).
{3 Registering a Deriver}
Once the generator function is defined, we can combine the argument extraction
and the generator function to create a
{{!Ppxlib.Deriving.Generator.t}[Deriving.Generator.t]}:
{@ocaml[
# let generator () = Deriving.Generator.V2.make (args()) generate_impl ;;
val generator : unit -> (structure_item list, 'a) Deriving.Generator.t = <abstr>
]}
This generator can then be registered as a deriver through the {{!Ppxlib.Deriving.add}[Deriving.add]}
function. Note that, {{!Ppxlib.Deriving.add}[Deriving.add]} will call {{!Ppxlib.Driver.register_transformation}[Driver.register_transformation]}
itself, so you won't need to do it manually. Adding a deriver is done in a way
that no two derivers with the same name can be registered. This includes derivers
registered through the {{:https://github.com/ocaml-ppx/ppx_deriving}ppx_deriving} library.
{@ocaml[
# let my_deriver = Deriving.add "my_deriver" ~str_type_decl:(generator()) ;;
val my_deriver : Deriving.t = <abstr>
]}
The different, optional named argument allows registering generators to be
applied in different contexts and in one function call. Remember that you
can only add one deriver with a given name, even if applied on different
contexts. As the API shows, derivers are restricted to being applied in the following contexts:
- Type declarations ([type t = Foo of int])
- Type extensions ([type t += Foo of int])
- Exceptions ([exception E of int])
- Module type declarations ([module type T = sig end])
in both structures and signatures.
{2 Attribute-guided Rewriting}
[ppxlib] provides context-free rules that, like derivers, apply to nodes based on their
attributes but, like extenders, allow rewriting the entire AST node. These provide
lighter-weight syntax than extenders but that also means it's less obvious that they're
rewriting the syntax tree.
Before using this kind of rule, carefully consider using an extender instead. [ppxlib]
provides an opinionated syntax for preprocessors so that it's easy for users to understand
what code is being affected by the PPX. In general, these should only be used to slightly
modify the node the attribute is attached to, rather than rewrite it to something new.
The syntax of extenders highlights to users where more involved rewriting is taking place.
These are composed of:
- The name of the rewrite rule
- The list of attributes they define
- The expand function
They are defined to apply in a specific context, specifically, they can be registered to
be processed in the same contexts as extenders can occur.
{3 The List of Attributes}
A given rewrite rule can have multiple attributes that trigger it, if any of the
attributes are present on a single node then the rule is triggered and provided with the
AST node along with the payload of all the attributes registered by this rule. To declare
attributes use the {{!Ppxlib.Attribute.declare}[Attribute.declare]} function (or the other
similar functions in that module). Note that the {{!Ppxlib.Attribute.Context}[Context.t]}
must match the type of AST nodes that the rule will apply to.
{@ocaml[
# let prefix_attr = Attribute.declare "example.prefix" Expression
Ast_pattern.(single_expr_payload (estring __)) Fun.id
and suffix_attr = Attribute.declare "example.suffix" Expression
Ast_pattern.(single_expr_payload (estring __)) Fun.id ;;
val prefix_attr : (expression, string) Attribute.t = <abstr>
val suffix_attr : (expression, string) Attribute.t = <abstr>
]}
{3 The Expand Function}
The expand function takes the AST node (with this rule's attributes already stripped) and
the payloads of all the declared attributes (as a list of [payload option] to allow for
attributes that haven't been included).
{@ocaml[
# let expander
~ctxt
expression
([ prefix; suffix ] : _ Context_free.Rule.Parsed_payload_list.t)
=
match expression.pexp_desc with
| Pexp_ident { txt = Lident name; loc } ->
let prefixed = Option.value ~default:"" prefix ^ name in
let suffixed = prefixed ^ Option.value ~default:"" suffix in
{ expression with pexp_desc = Pexp_ident { txt = Lident suffixed; loc } }
| _ -> expression ;;
val expander :
ctxt:'a ->
expression ->
(string * (string * unit)) Context_free.Rule.Parsed_payload_list.t -> expression =
<fun>
]}
{3 Creating a rewriting rule}
Finally, we can create the rule using the appropriate
{{!Ppxlib.Extension.Context}[Ppxlib.Extension.Context]} and register it with the driver using
{{!Ppxlib.Context_free.Rule.attr_multiple_replace}[Context_free.Rule.attr_multiple_replace]}.
There's also a {{!Ppxlib.Context_free.Rule.attr_replace}[Context_free.Rule.attr_replace]}
function with a slightly simpler API if you only use a single attribute.
{@ocaml[
# let rewrite_rule = Context_free.Rule.attr_multiple_replace "example" Expression
[ prefix_attr; suffix_attr ] expander ;;
val rule : Context_free.Rule.t = <abstr>
# Driver.register_transformation ~rules:[rewrite_rule] "example" ;;
- : unit = ()
]}
Now, for example, the following:
{@ocaml[
let _ = foo [@prefix "p_"] [@suffix "_s"]
]}
will be rewritten to:
{@ocaml[
let _ = p_foo_s
]}
{2 Constant Rewriting}
OCaml integrates a
{{:https://v2.ocaml.org/manual/extensionsyntax.html#ss:extension-literals}syntax}
to define special constants. Any [g..z] or [G..Z] suffix appended after a float
or int is accepted by the parser (but refused later by the compiler). This means
a PPX must rewrite them.
[ppxlib] provides the {{!Ppxlib.Context_free.Rule.constant}[Context_free.Rule.constant]} function to rewrite those litteral
constants. The character (between [g] and [z] or [G] and [Z]) has to be
provided, as well as the constant kind (float or int), and both the location and
the litteral as a string will be passed to a rewriting function:
{@ocaml[
# let kind = Context_free.Rule.Constant_kind.Integer ;;
val kind : Context_free.Rule.Constant_kind.t =
Ppxlib.Context_free.Rule.Constant_kind.Integer
# let rewriter loc s = Ast_builder.Default.eint ~loc (int_of_string s * 100) ;;
val rewriter : location -> string -> expression = <fun>
# let rule = Context_free.Rule.constant kind 'g' rewriter ;;
val rule : Context_free.Rule.t = <abstr>
# Driver.register_transformation ~rules:[ rule ] "constant" ;;
- : unit = ()
]}
As an example with the above transformation, [let x = 2g + 3g] will be
rewritten to [let x = 200 + 300].
{2 Special Functions}
[ppxlib] supports registering functions to be applied at compile time. A registered identifier [f_macro] will trigger rewriting in two situations:
{ol
{li When it plays the role of the function in a function application}
{li Anywhere it appears in an expression}
}
For instance, in
{@ocaml[
let _ = (f_macro arg1 arg2, f_macro)
]}
the rewriting will be triggered once for the left-hand side [f_macro arg1 arg2]
and once for the right hand side [f_macro]. It is the expansion function that
is responsible for distinguishing between the two cases: using pattern-matching to
distinguish between a function application in one case and a single identifier in
the other.
In order to register a special function, one needs to use {{!Ppxlib.Context_free.Rule.special_function}[Context_free.Rule.special_function]}, indicating the name of the special function and the rewriter. The rewriter will take the expression (without expansion context) and should output an [expression option], where:
- [None] signifies that no rewriting should be done: the top-down pass can
continue (potentially inside the expression).
- [Some exp] signifies the original expression should be replaced by [expr]. The
top-down pass continues with [expr].
The difference between [fun expr -> None] and [fun expr -> Some expr] is that
the former will continue the top-down pass {e inside} [expr], while the latter
will continue the top-down pass from [expr] (included), therefore starting an
infinite loop.
{@ocaml[
# let expand e =
let return n = Some (Ast_builder.Default.eint ~loc:e.pexp_loc n) in
match e.pexp_desc with
| Pexp_apply (_, arg_list) -> return (List.length arg_list)
| _ -> return 0
;;
val expand : expression -> expression option = <fun>
# let rule = Context_free.Rule.special_function "n_args" expand ;;
val rule : Context_free.Rule.t = <abstr>
# Driver.register_transformation ~rules:[ rule ] "special_function_demo" ;;
- : unit = ()
]}
With such a rewriter registered:
{[
# Printf.printf "n_args is applied with %d arguments\n" (n_args ignored "arguments");;
n_args is applied with 2 arguments
- : unit = ()
]}
{1:global_transformation Global transformation}
Global transformations are the most general kind of transformation. As such, they allow doing virtually any modifications, but this comes with several drawbacks. There are very few PPXs that really need this powerful but dangerous feature. In fact, even if, at first sight, it seems like your transformation isn't context-free, it's likely that you can find a more suitable abstraction with which it becomes context-free. Whenever that's the case, go for context-free! The mentioned drawbacks are:
- It is harder for the user to know exactly what parts of the AST will be
changed. Your transformation becomes a scary black box.
- It is harder for [ppxlib] to combine several global transformations, as there is no
guarantee that the effect of one will work well with the effect of another.
- The job done by two global transformations (e.g., an AST traverse) cannot be
factorised, resulting in slower compilation time.
- If you don't make sure that you really follow all {{!"good-practices"}good practices}, you might end up messing up the global developer experience.
For all these reasons, a global transformation should be avoided whenever a
context-free transformation could do the job, which by experience seems to be most of the time.
The API for defining a global transformation is easy. A global transformation consists simply of the function and can be directly be registered with {{!Ppxlib.Driver.register_transformation}[Driver.register_transformation]}.
{@ocaml[
# let f str = List.filter (fun _ -> Random.bool ()) str;; (* Randomly omit structure items *)
val f : 'a list -> 'a list = <fun>
# Driver.register_transformation ~impl:f "absent_minded_transformation"
- : unit = ()
]}
{1 Inlining Transformations}
When using a PPX, the transformation happens at compile time, and the produced code could be directly inlined into the original code. This allows dropping the dependency on [ppxlib] and the PPX used to generate the code.
This mechanism is implemented for derivers implemented in [ppxlib] and is convenient to use, especially in conjunction with Dune. When applying a deriver, using [[@@deriving_inline deriver_name]] will apply the inline mode of [deriver_name] instead of the normal mode.
Inline derivers will generate a [.corrected] version of the file that Dune can use to promote your file. For more information on how to use this feature to remove a dependency on [ppxlib] and a specific PPX from your project, refer to {{:https://ocaml.org/docs/metaprogramming#dropping-ppxs-dependency-with-derivinginline}this guide}.
In addition to [[@@deriving_inline]], there is also [[@@@expand_inline <structure payload>]] and [[@@@expand_inline: <signature payload>]]. These can be use to inline code generated by other context free transformations (not just derivers):
{@ocaml[
[@@@expand_inline let _ = [%add_suffix "foo"]]
let _ = "foo_suffixed"
[@@@end]
module type S = sig
[@@@expand_inline: type foo = [%pair_of string]]
type foo = string * string
[@@@end]
end
]}
{1 Integration with Dune}
If your PPX is written as a Dune project, you'll need to specify the [kind]
field in your [dune] file with one of the following two values:
- [ppx_rewriter], or
- [ppx_deriver].
If your transformation is anything but a deriver (e.g. an extension node rewriter), use [ppx_rewriter]. If your transformation is a deriver, then the TLDR workflow is: use [ppx_deriver] and furthermore add [ppx_deriving] to your dependencies, i.e. to the [libraries] field of your dune file. In fact, the situation is quite a bit more complex, though: apart from applying the registered transformations, the Ppxlib driver also does several checks. One of those consists in checking the following: whenever the source code contains [\[@@deriving foo (...)\]], then the Ppxlib driver expects a driver named [foo] to be registered. That's helpful to catch typos and missing dependencies on derivers and is certainly more hygienic than silently ignoring the annotation. However, for that check to work, the registered derivers must be grouped together into one process, i.e. a driver. UTop cannot use a static driver such as the Ppxlib one because dependencies are added dynamically to a UTop session. So the solution is the following: if you use [ppx_deriver] in your [kind] field, dune will add the right data to your PPXs META file to ensure that UTop will use the [ppx_deriving] driver, which links the derivers dynamically. As a result, [ppx_derivng] appears as a dependency in the META file. Therefore, whenever a user uses [ocamlfind] (e.g. by using UTop), they will hit an "[ppx_derivng] not found" error, unless you define [ppx_deriving] in your dependencies. So, long story short: if you strongly care about avoiding [ppx_deriving] as a dependency, use [ppx_rewriter] in your [kind] field and be aware of the fact that users won't be able to try your deriver in UTop; otherwise do the TLDR workflow.
Here is a minimal Dune stanza for a rewriter:
{@dune[
(library
(public_name my_ppx_rewriter)
(kind ppx_rewriter)
(libraries ppxlib))
]}
The public name you chose is the name your users will refer to your PPX in
the [preprocess] field. For example, to use this PPX rewriter, one would add the
[(preprocess (pps my_ppx_rewriter))] to their [library] or [executable] stanza.
{1:generatingcode Defining AST Transformations}
In this chapter, we only focused on the [ppxlib] ceremony to declare all kinds
of transformations. However, we did not cover how to write the actual
generative function, the backbone of the transformation. [ppxlib] provides several
modules to help with code generation and matching, which are covered in more
depth in the next chapters of this documentation:
- {{!Ppxlib.Ast_traverse}[Ast_traverse]}, which helps in defining AST traversals, such as maps, folds,
iter, etc.
- {{!Ppxlib.Ast_helper}[Ast_helper]} and {{!Ppxlib.Ast_builder}[Ast_builder]}, for generating AST nodes in a simpler way than
directly dealing with the {{!Ppxlib.Parsetree}[Parsetree]} types, providing a more stable API.
- {{!Ppxlib.Ast_pattern}[Ast_pattern]}, the sibling of {{!Ppxlib.Ast_builder}[Ast_builder]} for matching on AST nodes,
extracting values for them.
- {!Ppxlib_metaquot}, a PPX to manipulate code more simply by quoting and unquoting
code.
This documentation also includes some {{!"good-practices"}guidelines} on how to generate nice code.
We encourage you to read and follow it to produce high quality PPXs:
- A section on good {{!page-"good-practices"."handling_errors"}error reporting}
- A section on the {{!page-"good-practices"."quoting"}mechanism}
- A section on how to {{!page-"good-practices"."testing-your-ppx"}test} your PPX
- A section on how to collaborate with Merlin effectively by being careful with {{!page-"good-practices"."testing-your-ppx"}locations}
{%html: <div style="display: flex; justify-content:space-between"><div>%}{{!"driver"}< The Driver}{%html: </div><div>%}{{!"generating-code"}Generating AST nodes >}{%html: </div></div>%}