unikernel/duniverse/ppx_sexp_conv/README.org

@@ -0,0 +1,582 @@
+#+TITLE: ppx_sexp_conv
+
+* [@@deriving sexp]
+
+=ppx_sexp_conv= is a PPX syntax extension that generates code for
+converting OCaml types to and from s-expressions, as defined in the
+[[https://github.com/janestreet/sexplib][=sexplib=]] library.  S-expressions are defined by the following type:
+
+#+begin_src ocaml
+type sexp = Atom of string | List of sexp list
+#+end_src
+
+and are rendered as parenthesized lists of strings, /e.g./ =(This (is
+an) (s expression))=.
+
+=ppx_sexp_conv= fits into the [[https://github.com/whitequark/ppx_deriving][=ppx_deriving=]] framework, so you can
+invoke it the same way you invoke any other deriving plug-in.  Thus,
+we can write
+
+#+begin_src ocaml
+type int_pair = (int * int) [@@deriving sexp]
+#+end_src
+
+to get two values defined automatically, =sexp_of_int_pair= and
+=int_pair_of_sexp=.  If we only want one direction, we can write one
+of the following.
+
+#+begin_src ocaml
+type int_pair = (int * int) [@@deriving sexp_of]
+type int_pair = (int * int) [@@deriving of_sexp]
+#+end_src
+
+These sexp-converters depend on having a set of converters for basic
+values (/e.g./, =int_of_sexp=) already in scope.  This can be done by
+writing:
+
+#+begin_src ocaml
+open Sexplib.Std
+#+end_src
+
+If you're using [[https://github.com/janestreet/core][=Core=]], you can get the same effect with =open Core=.
+
+It's also possible to construct converters based on type expressions,
+/i.e./:
+
+#+begin_src ocaml
+  [%sexp_of: (int * string) list] [1,"one"; 2,"two"]
+  |> Sexp.to_string;;
+  => "((1 one) (2 two))"
+
+  [%sexp_of: (int * string) list] [1,"one"; 2,"two"]
+  |> [%of_sexp: (int * string) list];;
+  => [1,"one"; 2,"two"]
+#+end_src
+
+For =%sexp_of=, we can also omit the conversion of some types by
+putting underscores for that type name.
+
+#+begin_src ocaml
+  [%sexp_of: (int * _) list] [1,"one"; 2,"two"]
+  |> Sexp.to_string;;
+  => "((1 _)(2 _))"
+#+end_src
+
+* [@@deriving sexp_grammar]
+
+If =ppx_sexp_conv= can derive =of_sexp=, it can also generate a description of
+the sexps that the resulting =t_of_sexp= would accept.  This is the sexp grammar.
+See =Sexplib0.Sexp_grammar= for details. Use =[@@deriving sexp_grammar]= to derive
+the grammar for a type.
+
+It is possible to construct sexp grammars directly from type expressions, e.g.,
+
+#+BEGIN_SRC ocaml
+[%sexp_grammar: (int, bool array) Either.t Base.Map.M(String).t]
+#+END_SRC
+
+** Tagging grammars
+
+Use =[@sexp_grammar.tag key = value]=, where =(key : string)= and =(value :
+Sexp.t)=, to annotate a grammar with a tag that can be inspected at runtime.
+
+** Custom grammars
+
+Use =[@sexp_grammar.custom grammar]= to override a type's sexp grammar with
+=grammar=.
+
+** Stub grammars
+
+Annotate a type with =[@sexp_grammar.any]= to use a stub grammar that accepts
+any sexp. Alternately, write =[@sexp_grammar.any desc]= where =(desc : string)=
+to use =desc= as a human-readable description for the stub grammar.
+
+* Conversion rules
+
+In the following, we'll review the serialization rules for different
+OCaml types.
+
+** Basic types
+
+Basic types are represented as atoms.  For numbers like =int=,
+=int32=, =int64=, =float=, the string in the atom is what is accepted
+the standard ocaml functions =int_of_string=, =Int32.of_string=, etc.
+For the types =char= or =string=, the string in the atom is
+respectively a one character string or the string itself.
+
+** Lists and arrays
+
+OCaml-lists and arrays are represented as s-expression lists.
+
+** Tuples and unit
+
+OCaml tuples are treated as lists of values in the same order as in
+the tuple.  The type =unit= is treated like a 0-tuple.  /e.g./:
+
+#+begin_src ocaml
+  (3.14, "foo", "bar bla", 27)  =>  (3.14 foo "bar bla" 27)
+#+end_src
+
+** Options
+
+With options, =None= is treated as a zero-element list, and =Some= is
+treated as a singleton list, as shown below.
+
+#+begin_src ocaml
+None        =>  ()
+Some value  =>  (value)
+#+end_src
+
+We also support reading options following the ordinary rules for
+variants /i.e./:
+
+#+begin_src ocaml
+None        =>  None
+Some value  =>  (Some value)
+#+end_src
+
+The rules for variants are described below.
+
+** Records
+
+Records are represented as lists of lists, where each inner list is a
+key-value pair. Each pair consists of the name of the record field
+(first element), and its value (second element).  /e.g./:
+
+#+begin_src ocaml
+  { foo = (3,4);
+    bar = "some string"; }
+  => ((foo (3 4)) (bar "some string"))
+#+end_src
+
+Type specifications of records allow the use of several attributes. The
+attribute =sexp.option= indicates that a record field should be optional.
+/e.g./:
+
+#+begin_src ocaml
+  type t =
+    { x : int option;
+      y : int option [@sexp.option];
+    } [@@deriving sexp]
+#+end_src
+
+The following examples show how this works.
+
+#+begin_src ocaml
+  { x = Some 1; y = Some 2; } => ((x (1)) (y 2))
+  { x = None  ; y = None;   } => ((x ()))
+#+end_src
+
+Note that, when present, an optional value is represented as the bare
+value, rather than explicitly as an option.
+
+The attribute =sexp.bool= indicates that a boolean record field is shown
+as either present or absent, but not as containing a value.
+
+#+begin_src ocaml
+  type t = { enabled : bool [@sexp.bool] } [@@deriving sexp]
+
+  { enabled = true } => ((enabled))
+  { enabled = false } => ()
+#+end_src
+
+The attributes =sexp.list= and =sexp.array= indicate that a list or array record
+field, respectively, can be omitted when it is empty.
+
+#+begin_src ocaml
+  type t =
+    { arr : int array [@sexp.array]
+    ; lst : int list [@sexp.list]
+    }
+  [@@deriving sexp]
+
+  { arr = [||]; lst = [] } => ()
+  { arr = [|1;2|]; lst = [3;4] } => ((arr (1 2)) (lst (3 4)))
+#+end_src
+
+*** Defaults
+
+More complex default values can be specified explicitly using several
+constructs, /e.g./:
+
+#+begin_src ocaml
+  type t =
+    { a : int [@default 42];
+      b : int [@default 3] [@sexp_drop_default (=)];
+      c : int [@default 3] [@sexp_drop_if fun x -> x = 3];
+      d : int Queue.t [@sexp.omit_nil]
+    } [@@deriving sexp]
+#+end_src
+
+The =@default= annotation lets one specify a default value to be
+selected if the field is not specified, when converting from an
+s-expression.  The =@sexp_drop_default= annotation implies that the
+field will be dropped when generating the s-expression if the value
+being serialized is equal to the default according to the specified equality
+function. =@sexp_drop_if= is like =@sexp_drop_default=, except that
+it lets you specify the condition under which the field is dropped.
+Finally, =@sexp.omit_nil= means to treat a missing field as if it
+has value =List []= when reading, and drop the field if it has value
+=List []= when writing.
+
+**** Specifying equality for [@sexp_drop_default]
+
+The equality used by [@sexp_drop_default] is customizable. There
+are several ways to specify the equality function:
+
+#+begin_src ocaml
+  type t =
+    { a : u [@default u0] [@sexp_drop_default (=)]; (* explicit user-provided function *)
+      b : u [@default u0] [@sexp_drop_default.compare]; (* uses [%compare.equal: u] *)
+      c : u [@default u0] [@sexp_drop_default.equal]; (* uses [%equal: u] *)
+      d : u [@default u0] [@sexp_drop_default.sexp]; (* compares sexp representations *)
+      e : u [@default u0] [@sexp_drop_default]; (* deprecated. uses polymorphic equality. *)
+    } [@@deriving sexp]
+#+end_src
+
+*** Allowing extra fields
+
+The =@sexp.allow_extra_fields= annotation lets one specify that the
+sexp-converters should silently ignore extra fields, instead of
+raising.  This applies only to the record to which the annotation is
+attached, and not to deeper sexp converters that may be called during
+conversion of a sexp to the record.
+
+#+begin_src ocaml
+  type t = { a: int } [@@deriving sexp]
+  ((a 0)(b b)) => exception
+
+  type t = { a: int } [@@deriving sexp] [@@sexp.allow_extra_fields]
+  ((a 0)(b b)) => {a = 0}
+
+  type t = A of { a : int } [@sexp.allow_extra_fields] [@@deriving sexp]
+  (A (a 0)(b b)) => A {a = 0}
+#+end_src
+
+** Variants
+
+Constant constructors in variants are represented as
+strings. Constructors with arguments are represented as lists, the
+first element being the constructor name, the rest being its
+arguments. Constructors may also be started in lowercase in
+S-expressions, but will always be converted to uppercase when
+converting from OCaml values.
+
+For example:
+
+#+begin_src ocaml
+  type t = A | B of int * float * t [@@deriving sexp]
+  B (42, 3.14, B (-1, 2.72, A))  =>  (B 42 3.14 (B -1 2.72 A))
+#+end_src
+
+The above example also demonstrates recursion in data structures.
+
+Variants support the attribute =sexp.list= when a clause has a single
+list as its argument.
+
+#+begin_src ocaml
+  type t =
+   | A of int list
+   | B of int list [@sexp.list]
+
+  A [1; 2; 3] => (A (1 2 3))
+  B [1; 2; 3] => (B 1 2 3)
+#+end_src
+
+*** Inline records
+
+Constructors with inline records are represented as lists, the first element
+being the constructor name, the rest being the record fields, represented the
+same way as in record types, but without being wrapped in an extra layer of
+parentheses.
+
+#+begin_src ocaml
+  type t = A of { x : int }
+
+  A { x = 8 } => (A (x 8))
+#+end_src
+
+** Polymorphic variants
+
+Polymorphic variants behave almost the same as ordinary variants.  The
+notable difference is that polymorphic variant constructors must
+always start with an either lower- or uppercase character, matching
+the way it was specified in the type definition.  This is because
+OCaml distinguishes between upper and lowercase variant
+constructors. Note that type specifications containing unions of
+variant types are also supported by the S-expression converter, for
+example as in:
+
+#+begin_src ocaml
+  type ab = [ `A | `B ] [@@deriving sexp]
+  type cd = [ `C | `D ] [@@deriving sexp]
+  type abcd = [ ab | cd ] [@@deriving sexp]
+#+end_src
+
+However, because `ppx_sexp_conv` needs to generate additional code to
+support inclusions of polymorphic variants, `ppx_sexp_conv` needs to
+know when processing a type definition whether it might be included in
+a polymorphic variant. `ppx_sexp_conv` will only generate the extra
+code automatically in the common case where the type definition is
+syntactically a polymorphic variant like in the example
+above. Otherwise, you will need to indicate it by using `[@@deriving
+sexp_poly]` (resp `of_sexp_poly`) instead of `[@@deriving sexp]` (resp
+`of_sexp`):
+
+#+begin_src ocaml
+  type ab = [ `A | `B ] [@@deriving sexp]
+  type alias_of_ab = ab [@@deriving sexp_poly]
+  type abcd = [ ab | `C | `D ] [@@deriving sexp]
+#+end_src
+
+** Polymorphic values
+
+There is nothing special about polymorphic values as long as there are
+conversion functions for the type parameters.  /e.g./:
+
+#+begin_src ocaml
+type 'a t = A | B of 'a [@@deriving sexp]
+type foo = int t [@@deriving sexp]
+#+end_src
+
+In the above case the conversion functions will behave as if =foo= had
+been defined as a monomorphic version of =t= with ='a= replaced by
+=int= on the right hand side.
+
+If a data structure is indeed polymorphic and you want to convert it,
+you will have to supply the conversion functions for the type
+parameters at runtime.  If you wanted to convert a value of type ='a
+t= as in the above example, you would have to write something like
+this:
+
+#+begin_src ocaml
+  sexp_of_t sexp_of_a v
+#+end_src
+
+where =sexp_of_a=, which may also be named differently in this
+particular case, is a function that converts values of type ='a= to an
+S-expression.  Types with more than one parameter require passing
+conversion functions for those parameters in the order of their
+appearance on the left hand side of the type definition.
+
+** Opaque values
+
+Opaque values are ones for which we do not want to perform
+conversions.  This may be, because we do not have S-expression
+converters for them, or because we do not want to apply them in a
+particular type context. /e.g./ to hide large, unimportant parts of
+configurations.  To prevent the preprocessor from generating calls to
+converters, simply apply the attribute =sexp.opaque= to the type.  If the type
+is for a record field, it will likely need parentheses to avoid applying the
+attribute to the record field itself, /e.g./:
+
+#+begin_src ocaml
+  type foo = int * (stuff [@sexp.opaque]) [@@deriving sexp]
+
+  type bar =
+    { a : int
+    ; b : (stuff [@sexp.opaque])
+    }
+  [@@deriving sexp]
+#+end_src
+
+Thus, there is no need to specify converters for type =stuff=, and if
+there are any, they will not be used in this particular context.
+Needless to say, it is not possible to convert such an S-expression
+back to the original value.  Here is an example conversion:
+
+#+begin_src ocaml
+  (42, some_stuff)  =>  (42 <opaque>)
+#+end_src
+
+** Exceptions
+
+S-expression converters for exceptions can be automatically
+registered.
+
+#+begin_src ocaml
+  module M = struct
+    exception Foo of int [@@deriving sexp]
+  end
+#+end_src
+
+Such exceptions will be translated in a similar way as sum types, but
+their constructor will be prefixed with the fully qualified module
+path (here: =M.Foo=) so as to be able to discriminate between them
+without problems.
+
+The user can then easily convert an exception matching the above one
+to an S-expression using =sexp_of_exn=.  User-defined conversion
+functions can be registered, too, by calling =add_exn_converter=.
+This should make it very convenient for users to catch arbitrary
+exceptions escaping their program and pretty-printing them, including
+all arguments, as S-expressions.  The library already contains
+mappings for all known exceptions that can escape functions in the
+OCaml standard library.
+
+** Hash tables
+
+The Stdlib's Hash tables, which are abstract values in OCaml, are
+represented as association lists, /i.e./ lists of key-value pairs,
+/e.g./:
+
+#+begin_src scheme
+  ((foo 42) (bar 3))
+#+end_src
+
+Reading in the above S-expression as hash table mapping strings to
+integers (=(string, int) Hashtbl.t=) will map =foo= to =42= and =bar=
+to =3=.
+
+Note that the order of elements in the list may matter, because the
+OCaml-implementation of hash tables keeps duplicates.  Bindings will
+be inserted into the hash table in the order of appearance. Therefore,
+the last binding of a key will be the "visible" one, the others are
+"hidden".  See the OCaml documentation on hash tables for details.
+
+* A note about signatures
+
+In signatures, =ppx_sexp_conv= tries to generate an include of a named
+interface, instead of a list of value bindings.
+That is:
+
+#+begin_src ocaml
+type 'a t [@@deriving sexp]
+#+end_src
+
+will generate:
+
+#+begin_src ocaml
+include Sexpable.S1 with type 'a t := 'a t
+#+end_src
+
+instead of:
+
+#+begin_src ocaml
+val t_of_sexp : (Sexp.t -> 'a) -> Sexp.t -> 'a t
+val sexp_of_t : ('a -> Sexp.t) -> 'a t -> Sexp.t
+#+end_src
+
+There are however a number of limitations:
+- the type has to be named t
+- the type can only have up to 3 parameters
+- there shouldn't be any constraint on the type parameters
+
+If these aren't met, then =ppx_sexp_conv= will simply generate a list of value
+bindings.
+
+** Weird looking type errors
+
+In some cases, a type can meet all the conditions listed above, in which case the
+rewriting will apply, but lead to a type error. This happens when the type [t]
+is an alias to a type which does have constraints on the parameters, for
+instance:
+
+#+begin_src ocaml
+type 'a s constraint 'a = [> `read ]
+val sexp_of_s : ...
+val s_of_sexp : ...
+type 'a t = 'a s [@@deriving_inline sexp]
+include Sexpable.S1 with type 'a t := 'a t
+[@@@end]
+#+end_src
+
+will give an error looking like:
+
+#+begin_src
+Error: In this `with' constraint, the new definition of t
+       does not match its original definition in the constrained signature:
+       Type declarations do not match:
+         type 'a t = 'a t constraint 'a = [> `read ]
+       is not included in
+         type 'a t
+       File "sexpable.mli", line 8, characters 21-58: Expected declaration
+       Their constraints differ.
+#+end_src
+
+To workaround that error, simply copy the constraint on the type which has the
+=[@@deriving]= annotation. This will force generating a list of value bindings.
+
+* Deprecated syntax
+
+Originally, ~ppx_sexp_conv~ used special types instead of attributes. Those
+types have been replaced with attributes. Here are the appropriate conversions
+to update from code using the old types to the new attributes.
+
+
+** Opaque types
+
+Convert uses of ~sexp_opaque~ to uses of ~[@sexp.opaque]~. The ~[@sexp.opaque]~
+attribute usually needs explicit parentheses to clarify what type it annotate.
+
+Before:
+
+#+begin_src ocaml
+type t = int sexp_opaque list
+[@@deriving sexp]
+#+end_src
+
+After:
+
+#+begin_src ocaml
+type t = (int [@sexp.opaque]) list
+[@@deriving sexp]
+#+end_src
+
+** Record fields
+
+Convert uses of ~sexp_option~, ~sexp_list~, ~sexp_array~, and ~sexp_bool~ to
+uses of ~[@sexp.option]~, ~[@sexp.list]~, ~[@sexp.array]~, and ~[@sexp.bool]~ as
+appropriate. The attribute only specifies the modification, not the type, so you
+will need to use the regular types ~option~, ~list~, ~array~, and/or ~bool~ as
+well. Unlike ~[@sexp.opaque]~, these attributes do not need extra parentheses.
+
+Before:
+
+#+begin_src ocaml
+type t =
+  { a : int sexp_option
+  ; b : int sexp_list
+  ; c : int sexp_array
+  ; d : sexp_bool
+  }
+[@@deriving sexp]
+#+end_src
+
+After:
+
+#+begin_src ocaml
+type t =
+  { a : int option [@sexp.option]
+  ; b : int list [@sexp.list]
+  ; c : int array [@sexp.array]
+  ; d : bool [@sexp.bool]
+  }
+[@@deriving sexp]
+#+end_src
+
+** Variant constructors
+
+Convert uses of ~sexp_list~ in variants and polymorphic variants to uses of
+~[@sexp.list]~. You need to add the regular type ~list~ as well. Unlike
+~[@sexp.opaque]~, this attribute does not need extra parentheses.
+
+Before:
+
+#+begin_src ocaml
+type t = A of int sexp_list
+[@@deriving sexp]
+
+type u = [`B of int sexp_list]
+[@@deriving sexp]
+#+end_src
+
+After:
+
+#+begin_src ocaml
+type t = A of int list [@sexp.list]
+[@@deriving sexp]
+
+type u = [`B of int list [@sexp.list]]
+[@@deriving sexp]
+#+end_src