diff options
author | Stephane Glondu <steph@glondu.net> | 2020-01-31 11:38:07 +0100 |
---|---|---|
committer | Stéphane Glondu <steph@glondu.net> | 2020-01-31 11:38:07 +0100 |
commit | f20e2fd2753517f8931c478d13817ec85f2cc2e6 (patch) | |
tree | a3b172e78f81c57460a97d5af2c66709f7a49057 /doc | |
parent | 259a3a4f6378740a62c6d4de1d553b5f6c24f152 (diff) |
New upstream version 2.1.0
Diffstat (limited to 'doc')
37 files changed, 2756 insertions, 81 deletions
diff --git a/doc/atdgen.rst b/doc/atdgen.rst new file mode 100644 index 0000000..7b381c2 --- /dev/null +++ b/doc/atdgen.rst @@ -0,0 +1,1644 @@ +**************** +Atdgen reference +**************** + +Description +=========== + +Atdgen is a command-line program that takes as input type definitions in +the `ATD syntax <http://mjambon.com/atd>`__ and produces OCaml code +suitable for data serialization and deserialization. + +Two data formats are currently supported, these are +`JSON <http://json.org/>`__ and +`biniou <http://mjambon.com/biniou.html>`__, a binary format with +extensibility properties similar to JSON. Atdgen-json and Atdgen-biniou +will refer to Atdgen used in one context or the other. + +Atdgen was designed with efficiency and durability in mind. Software +authors are encouraged to use Atdgen directly and to write tools that +may reuse part of Atdgen's source code. + +Atdgen uses the following packages that were developed in conjunction +with Atdgen: + +- ``atd``: parser for the syntax of type definitions +- ``biniou``: parser and printer for biniou, a binary extensible data + format +- ```yojson`` <http://mjambon.com/yojson.html>`__: parser and printer + for JSON, a widespread text-based data format + +Command-line usage +================== + +Command-line help +----------------- + +Call ``atdgen -help`` for the full list of available options. + +Atdgen-json example +------------------- + +:: + + $ atdgen -t example.atd + $ atdgen -j -j-std example.atd + +Input file ``example.atd``: + +.. code:: ocaml + + type profile = { + id : string; + email : string; + ~email_validated : bool; + name : string; + ?real_name : string option; + ~about_me : string list; + ?gender : gender option; + ?date_of_birth : date option; + } + + type gender = [ Female | Male ] + + type date = { + year : int; + month : int; + day : int; + } + +is used to produce files ``example_t.mli``, ``example_t.ml``, +``example_j.mli`` and ``example_j.ml``. This is ``example_j.mli``: + +.. code:: ocaml + + (* Auto-generated from "example.atd" *) + + + type gender = Example_t.gender + + type date = Example_t.date = { year: int; month: int; day: int } + + type profile = Example_t.profile = { + id: string; + email: string; + email_validated: bool; + name: string; + real_name: string option; + about_me: string list; + gender: gender option; + date_of_birth: date option + } + + val write_gender : + Bi_outbuf.t -> gender -> unit + (** Output a JSON value of type {!gender}. *) + + val string_of_gender : + ?len:int -> gender -> string + (** Serialize a value of type {!gender} + into a JSON string. + @param len specifies the initial length + of the buffer used internally. + Default: 1024. *) + + val read_gender : + Yojson.Safe.lexer_state -> Lexing.lexbuf -> gender + (** Input JSON data of type {!gender}. *) + + val gender_of_string : + string -> gender + (** Deserialize JSON data of type {!gender}. *) + + val write_date : + Bi_outbuf.t -> date -> unit + (** Output a JSON value of type {!date}. *) + + val string_of_date : + ?len:int -> date -> string + (** Serialize a value of type {!date} + into a JSON string. + @param len specifies the initial length + of the buffer used internally. + Default: 1024. *) + + val read_date : + Yojson.Safe.lexer_state -> Lexing.lexbuf -> date + (** Input JSON data of type {!date}. *) + + val date_of_string : + string -> date + (** Deserialize JSON data of type {!date}. *) + + val write_profile : + Bi_outbuf.t -> profile -> unit + (** Output a JSON value of type {!profile}. *) + + val string_of_profile : + ?len:int -> profile -> string + (** Serialize a value of type {!profile} + into a JSON string. + @param len specifies the initial length + of the buffer used internally. + Default: 1024. *) + + val read_profile : + Yojson.Safe.lexer_state -> Lexing.lexbuf -> profile + (** Input JSON data of type {!profile}. *) + + val profile_of_string : + string -> profile + (** Deserialize JSON data of type {!profile}. *) + +Module ``Example_t`` (files ``example_t.mli`` and ``example_t.ml``) +contains all OCaml type definitions that can be used independently from +Biniou or JSON. + +For convenience, these definitions are also made available from the +``Example_j`` module whose interface is shown above. Any type name, +record field name or variant constructor can be referred to using either +module. For example, the OCaml expressions +``((x : Example_t.date) : Example_j.date)`` and +``x.Example_t.year = x.Example_j.year`` are both valid. + +Atdgen-biniou example +--------------------- + +:: + + $ atdgen -t example.atd + $ atdgen -b example.atd + +Input file ``example.atd``: + +.. code:: ocaml + + type profile = { + id : string; + email : string; + ~email_validated : bool; + name : string; + ?real_name : string option; + ~about_me : string list; + ?gender : gender option; + ?date_of_birth : date option; + } + + type gender = [ Female | Male ] + + type date = { + year : int; + month : int; + day : int; + } + +is used to produce files ``example_t.mli``, ``example_t.ml``, +``example_b.mli`` and ``example_b.ml``. + +This is ``example_b.mli``: + +.. code:: ocaml + + (* Auto-generated from "example.atd" *) + + + type gender = Example_t.gender + + type date = Example_t.date = { year: int; month: int; day: int } + + type profile = Example_t.profile = { + id: string; + email: string; + email_validated: bool; + name: string; + real_name: string option; + about_me: string list; + gender: gender option; + date_of_birth: date option + } + + (* Writers for type gender *) + + val gender_tag : Bi_io.node_tag + (** Tag used by the writers for type {!gender}. + Readers may support more than just this tag. *) + + val write_untagged_gender : + Bi_outbuf.t -> gender -> unit + (** Output an untagged biniou value of type {!gender}. *) + + val write_gender : + Bi_outbuf.t -> gender -> unit + (** Output a biniou value of type {!gender}. *) + + val string_of_gender : + ?len:int -> gender -> string + (** Serialize a value of type {!gender} into + a biniou string. *) + + (* Readers for type gender *) + + val get_gender_reader : + Bi_io.node_tag -> (Bi_inbuf.t -> gender) + (** Return a function that reads an untagged + biniou value of type {!gender}. *) + + val read_gender : + Bi_inbuf.t -> gender + (** Input a tagged biniou value of type {!gender}. *) + + val gender_of_string : + ?pos:int -> string -> gender + (** Deserialize a biniou value of type {!gender}. + @param pos specifies the position where + reading starts. Default: 0. *) + + (* Writers for type date *) + + val date_tag : Bi_io.node_tag + (** Tag used by the writers for type {!date}. + Readers may support more than just this tag. *) + + val write_untagged_date : + Bi_outbuf.t -> date -> unit + (** Output an untagged biniou value of type {!date}. *) + + val write_date : + Bi_outbuf.t -> date -> unit + (** Output a biniou value of type {!date}. *) + + val string_of_date : + ?len:int -> date -> string + (** Serialize a value of type {!date} into + a biniou string. *) + + (* Readers for type date *) + + val get_date_reader : + Bi_io.node_tag -> (Bi_inbuf.t -> date) + (** Return a function that reads an untagged + biniou value of type {!date}. *) + + val read_date : + Bi_inbuf.t -> date + (** Input a tagged biniou value of type {!date}. *) + + val date_of_string : + ?pos:int -> string -> date + (** Deserialize a biniou value of type {!date}. + @param pos specifies the position where + reading starts. Default: 0. *) + + (* Writers for type profile *) + + val profile_tag : Bi_io.node_tag + (** Tag used by the writers for type {!profile}. + Readers may support more than just this tag. *) + + val write_untagged_profile : + Bi_outbuf.t -> profile -> unit + (** Output an untagged biniou value of type {!profile}. *) + + val write_profile : + Bi_outbuf.t -> profile -> unit + (** Output a biniou value of type {!profile}. *) + + val string_of_profile : + ?len:int -> profile -> string + (** Serialize a value of type {!profile} into + a biniou string. *) + + (* Readers for type profile *) + + val get_profile_reader : + Bi_io.node_tag -> (Bi_inbuf.t -> profile) + (** Return a function that reads an untagged + biniou value of type {!profile}. *) + + val read_profile : + Bi_inbuf.t -> profile + (** Input a tagged biniou value of type {!profile}. *) + + val profile_of_string : + ?pos:int -> string -> profile + (** Deserialize a biniou value of type {!profile}. + @param pos specifies the position where + reading starts. Default: 0. *) + +Module ``Example_t`` (files ``example_t.mli`` and ``example_t.ml``) +contains all OCaml type definitions that can be used independently from +Biniou or JSON. + +For convenience, these definitions are also made available from the +``Example_b`` module whose interface is shown above. Any type name, +record field name or variant constructor can be referred to using either +module. For example, the OCaml expressions +``((x : Example_t.date) : Example_b.date)`` and +``x.Example_t.year = x.Example_b.year`` are both valid. + +Validator example +----------------- + +:: + + $ atdgen -t example.atd + $ atdgen -v example.atd + +Input file ``example.atd``: + +.. code:: ocaml + + type month = int <ocaml valid="fun x -> x >= 1 && x <= 12"> + type day = int <ocaml valid="fun x -> x >= 1 && x <= 31"> + + type date = { + year : int; + month : month; + day : day; + } + <ocaml validator="Date_util.validate_date"> + +is used to produce files ``example_t.mli``, ``example_t.ml``, +``example_v.mli`` and ``example_v.ml``. This is ``example_v.ml``, +showing how the user-specified validators are used: + +.. code:: ocaml + + (* Auto-generated from "example.atd" *) + + + type gender = Example_t.gender + + type date = Example_t.date = { year: int; month: int; day: int } + + type profile = Example_t.profile = { + id: string; + email: string; + email_validated: bool; + name: string; + real_name: string option; + about_me: string list; + gender: gender option; + date_of_birth: date option + } + + val validate_gender : + Atdgen_runtime.Util.Validation.path -> gender -> Atdgen_runtime.Util.Validation.error option + (** Validate a value of type {!gender}. *) + + val create_date : + year: int -> + month: int -> + day: int -> + unit -> date + (** Create a record of type {!date}. *) + + val validate_date : + Atdgen_runtime.Util.Validation.path -> date -> Atdgen_runtime.Util.Validation.error option + (** Validate a value of type {!date}. *) + + val create_profile : + id: string -> + email: string -> + ?email_validated: bool -> + name: string -> + ?real_name: string -> + ?about_me: string list -> + ?gender: gender -> + ?date_of_birth: date -> + unit -> profile + (** Create a record of type {!profile}. *) + + val validate_profile : + Atdgen_runtime.Util.Validation.path -> profile -> Atdgen_runtime.Util.Validation.error option + (** Validate a value of type {!profile}. *) + +Default type mapping +==================== + +The following table summarizes the default mapping between ATD types and +OCaml, biniou and JSON data types. For each language more +representations are available and are detailed in the next section of +this manual. + ++-----------------+---------------------+-------------------+--------------------+ +| ATD | OCaml | JSON | Biniou | ++=================+=====================+===================+====================+ +| ``unit`` | ``unit`` | null | unit | ++-----------------+---------------------+-------------------+--------------------+ +| ``bool`` | ``bool`` | boolean | bool | ++-----------------+---------------------+-------------------+--------------------+ +| ``int`` | ``int`` | -?(0\|[1-9][0-9]\ | svint | +| | | *) | | ++-----------------+---------------------+-------------------+--------------------+ +| ``float`` | ``float`` | number | float64 | ++-----------------+---------------------+-------------------+--------------------+ +| ``string`` | ``string`` | string | string | ++-----------------+---------------------+-------------------+--------------------+ +| ``'a option`` | ``'a option`` | ``"None"`` or | numeric variants | +| | | ``["Some", ...]`` | (tag 0) | ++-----------------+---------------------+-------------------+--------------------+ +| ``'a nullable`` | ``'a option`` | ``null`` or | numeric variants | +| | | representation of | (tag 0) | +| | | ``'a`` | | ++-----------------+---------------------+-------------------+--------------------+ +| ``'a list`` | ``'a list`` | array | array | ++-----------------+---------------------+-------------------+--------------------+ +| ``'a shared`` | no wrapping | not implemented | no longer | +| | | | supported | ++-----------------+---------------------+-------------------+--------------------+ +| ``'a wrap`` | defined by | representation of | representation of | +| | annotation, | ``'a`` | ``'a`` | +| | converted from | | | +| | ``'a`` | | | ++-----------------+---------------------+-------------------+--------------------+ +| variants | polymorphic | variants | regular variants | +| | variants | | | ++-----------------+---------------------+-------------------+--------------------+ +| record | record | object | record | ++-----------------+---------------------+-------------------+--------------------+ +| ``('a * 'b)`` | ``('a * 'b)`` | array | tuple | ++-----------------+---------------------+-------------------+--------------------+ +| ``('a)`` | ``'a`` | array | tuple | ++-----------------+---------------------+-------------------+--------------------+ + +Notes: + +- Null JSON fields by default are treated as if the field was missing. + They can be made meaningful with the ``keep_nulls`` flag. +- JSON nulls are used to represent the unit value and is useful for + instanciating parametrized types with "nothing". +- OCaml floats are written to JSON numbers with either a decimal point + or an exponent such that they are distinguishable from ints, even + though the JSON standard does not require a distinction between the + two. +- The optional values of record fields denoted in ATD by a question + mark are unwrapped or omitted in both biniou and JSON. +- JSON option values and JSON variants are represented in standard JSON + (``atdgen -j -j-std``) by a single string e.g. ``"None"`` or a pair + in which the first element is the name (constructor) e.g. + ``["Some", 1234]``. Yojson also provides a specific syntax for + variants using edgy brackets: ``<"None">``, ``<"Some": 1234>``. +- Biniou field names and variant names other than the option types use + the hash of the ATD field or variant name and cannot currently be + overridden by annotations. +- JSON tuples in standard JSON (``atdgen -j -j-std``) use the array + notation e.g. ``["ABC", 123]``. Yojson also provides a specific + syntax for tuples using parentheses, e.g. ``("ABC", 123)``. +- Types defined as abstract are defined in another module. + +ATD Annotations +=============== + +Section ``json`` +------------------ + +Field ``keep_nulls`` +~~~~~~~~~~~~~~~~~~~~~~ + +Position: after record + +Values: none, ``true`` or ``false`` + +Semantics: this flag, if present or set to true, indicates that fields +whose JSON value is ``null`` should not be treated as if they were +missing. In this case, ``null`` is parsed as a normal value, possibly of +a ``nullable`` type. + +Example: patch semantics + +.. code:: ocaml + + (* Type of the objects stored in our database *) + type t = { + ?x : int option; + ?y : int option; + ?z : int option; + } + +.. code:: ocaml + + (* Type of the requests to modify some of the fields of an object. *) + type t_patch = { + ?x : int nullable option; (* OCaml type: int option option *) + ?y : int nullable option; + ?z : int nullable option; + } <ocaml field_prefix="patch_"> <json keep_nulls> + +Let's consider the following json patch that means "set ``x`` to 1, +clear ``y`` and keep ``z`` as it is": + +:: + + { + "x": 1, + "y": null + } + +It will be parsed by the generated function ``t_patch_of_string`` into +the following OCaml value: + +.. code:: ocaml + + { + patch_x = Some (Some 1); + patch_y = Some None; + patch_z = None; + } + +Then presumably some code would be written to apply the patch to an +object of type ``t``. Such code is not generated by atdgen at this time. + +Available: from atd 1.12 + +Field ``name`` +~~~~~~~~~~~~~~~~ + +Position: after field name or variant name + +Values: any string making a valid JSON string value + +Semantics: specifies an alternate object field name or variant name to +be used by the JSON representation. + +Example: + +.. code:: ocaml + + type color = [ + Black <json name="black"> + | White <json name="white"> + | Grey <json name="grey"> + ] + + type profile = { + id <json name="ID"> : int; + username : string; + background_color : color; + } + +A valid JSON object of the ``profile`` type above is: + +:: + + { + "ID": 12345678, + "username": "kimforever", + "background_color": "black" + } + +Field ``repr`` +~~~~~~~~~~~~~~~~ + +Association lists +^^^^^^^^^^^^^^^^^ + +Position: after ``(string * _) list`` type + +Values: ``object`` + +Semantics: uses JSON's object notation to represent association lists. + +Example: + +.. code:: ocaml + + type counts = (string * int) list <json repr="object"> + +A valid JSON object of the ``counts`` type above is: + +:: + + { + "bob": 3, + "john": 1408, + "mary": 450987, + "peter": 93087 + } + +Without the annotation ``<json repr="object">``, the data above would be +represented as: + +:: + + [ + [ "bob", 3 ], + [ "john", 1408 ], + [ "mary", 450987 ], + [ "peter", 93087 ] + ] + +Floats +^^^^^^ + +Position: after ``float`` type + +Values: ``int`` + +Semantics: specifies a float value that must be rounded to the nearest +integer and represented in JSON without a decimal point nor an exponent. + +Example: + +.. code:: ocaml + + type unixtime = float <json repr="int"> + +Field ``tag_field`` +~~~~~~~~~~~~~~~~~~~~~ + +Superseded by ``<json adapter.ocaml="...">``. Available since atdgen +1.5.0 and yojson 1.2.0 until atdgen 1.13. + +This feature makes it possible to read JSON objects representing +variants that use one field for the tag and another field for the +untagged value of the specific type associated with that tag. + +Position: on a record field name, for a field holding a variant type. + +Value: name of another JSON field which holds the string representing +the constructor for the variant. + +Semantics: The type definition + +.. code:: ocaml + + type t = { + value <json tag_field="kind">: [ A | B <json name="b"> of int ]; + } + +covers JSON objects that have an extra field ``kind`` which holds either +``"A"`` or ``"b"``. Valid JSON values of type ``t`` include +``{ "kind": "A" }`` and ``{ "kind": "b", "value": 123 }``. + +Field ``untyped`` +~~~~~~~~~~~~~~~~~~~ + +Superseded by ``<json open_enum>`` and ``<json adapter.ocaml="...">``. +Available since atdgen 1.10.0 and atd 1.2.0 until atdgen 1.13. + +This flag enables parsing of arbitrary variants without prior knowledge +of their type. It is useful for constructing flexible parsers for +extensible serializations. ``json untyped`` is compatible with regular +variants, ``json tag_field`` variants, default values, and implicit +``tag_field`` constructors. + +Position: on a variant constructor with argument type +``string * json option`` (at most one per variant type) + +Value: none, ``true`` or ``false`` + +Semantics: The type definition + +.. code:: ocaml + + type v = [ + | A + | B <json name="b"> of int + | Unknown <json untyped> of (string * json option) + ] + +will parse and print ``"A"``, ``["b", 0]``, ``"foo"``, and +``["bar", [null]]`` in a regular variant context. In the ``tag_field`` +type ``t`` context in the previous section, ``v`` will parse and print +``{ "kind": "foo" }`` and ``{ "kind": "bar", "value": [null] }`` as well +as the examples previously given. + +Field ``open_enum`` +~~~~~~~~~~~~~~~~~~~~~ + +Where an enum (finite set of strings) is expected, this flag allows +unexpected strings to be kept under a catch-all constructor rather than +producing an error. + +Position: on a variant type comprising exactly one constructor with an +argument. The type of that argument must be ``string``. All other +constructors must have no arguments. + +Value: none + +For example: + +.. code:: ocaml + + type language = [ + | English + | Chinese + | Other of string + ] <json open_enum> + +maps the json string ``"Chinese"`` to the OCaml value ```Chinese`` and +maps ``"French"`` to ```Other "French"``. + +Available since atdgen 2.0. + +Field ``adapter.ocaml`` +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Json adapters are a mechanism for rearranging json data on-the-fly, so +as to make them compatible with ATD. The programmer must provide an +OCaml module that provides converters between the original json +representation and the ATD-compatible representation. The signature of +the user-provided module must be equal to +``Atdgen_runtime.Json_adapter.S``, which is: + +.. code:: ocaml + + sig + (** Convert from original json to ATD-compatible json *) + val normalize : Yojson.Safe.t -> Yojson.Safe.t + + (** Convert from ATD-compatible json to original json *) + val restore : Yojson.Safe.t -> Yojson.Safe.t + end + +The type ``Yojson.Safe.t`` is the type of parsed JSON as provided by +the yojson library. + +Position: on a variant type or on a record type. + +Value: an OCaml module identifier. Note that +``Atdgen_runtime.Json_adapter`` provides a few modules and functors that +are ready to use. Users are however encouraged to write their own to +suit their needs. + +Sample ATD definitions: + +.. code:: ocaml + + type document = [ + | Image of image + | Text of text + ] <json adapter.ocaml="Atdgen_runtime.Json_adapter.Type_field"> + + type image = { + url: string; + } + + type text = { + title: string; + body: string; + } + +ATD-compliant json values: + +- ``["Image", {"url": "https://example.com/ocean123.jpg"}]`` +- ``["Text", {"title": "Cheeses Around the World", "body": "..."}]`` + +Corresponding json values given by some API: + +- ``{"type": "Image", "url": "https://example.com/ocean123.jpg"}`` +- ``{"type": "Text", "title": "Cheeses Around the World", "body": "..."}`` + +The json adapter ``Type_field`` that ships with the atdgen runtime takes +care of converting between these two forms. For information on how to +write your own adapter, please consult the documentation for the yojson +library. + +Section ``biniou`` +-------------------- + +Field ``repr`` +~~~~~~~~~~~~~~~~ + +Integers +^^^^^^^^ + +Position: after ``int`` type + +Values: ``svint`` (default), ``uvint``, ``int8``, ``int16``, ``int32``, +``int64`` + +Semantics: specifies an alternate type for representing integers. The +default type is ``svint``. The other integers types provided by biniou +are supported by Atdgen-biniou. They have to map to the corresponding +OCaml types in accordance with the following table: + ++---------------+------------------------+---------------------------------------+ +| Biniou type | Supported OCaml type | OCaml value range | ++===============+========================+=======================================+ +| ``svint`` | ``int`` | ``min_int`` ... ``max_int`` | ++---------------+------------------------+---------------------------------------+ +| ``uvint`` | ``int`` | 0 ... ``max_int``, ``min_int`` ... -1 | ++---------------+------------------------+---------------------------------------+ +| ``int8`` | ``char`` | ``'\000`` ... ``'\255`` | ++---------------+------------------------+---------------------------------------+ +| ``int16`` | ``int`` | 0 ... 65535 | ++---------------+------------------------+---------------------------------------+ +| ``int32`` | ``int32`` | ``Int32.min_int`` ... | +| | | ``Int32.max_int`` | ++---------------+------------------------+---------------------------------------+ +| ``int64`` | ``int64`` | ``Int64.min_int`` ... | +| | | ``Int64.max_int`` | ++---------------+------------------------+---------------------------------------+ + +In addition to the mapping above, if the OCaml type is ``int``, any +biniou integer type can be read into OCaml data regardless of the +declared biniou type. + +Example: + +.. code:: ocaml + + type t = { + id : int + <ocaml repr="int64"> + <biniou repr="int64">; + data : string list; + } + +Floating-point numbers +^^^^^^^^^^^^^^^^^^^^^^ + +Position: after ``float`` type + +Values: ``float64`` (default), ``float32`` + +Semantics: ``float32`` allows for a shorter serialized representation of +floats, using 4 bytes instead of 8, with reduced precision. OCaml floats +always use 8 bytes, though. + +Example: + +.. code:: ocaml + + type t = { + lat : float <biniou repr="float32">; + lon : float <biniou repr="float32">; + } + +Arrays and tables +^^^^^^^^^^^^^^^^^ + +Position: applies to lists of records + +Values: ``array`` (default), ``table`` + +Semantics: ``table`` uses biniou's table format instead of a regular +array for serializing OCaml data into biniou. Both formats are supported +for reading into OCaml data regardless of the annotation. The table +format allows + +Example: + +.. code:: ocaml + + type item = { + id : int; + data : string list; + } + + type items = item list <biniou repr="table"> + +Section ``ocaml`` +------------------- + +Field ``predef`` +~~~~~~~~~~~~~~~~~~ + +Position: left-hand side of a type definition, after the type name + +Values: none, ``true`` or ``false`` + +Semantics: this flag indicates that the corresponding OCaml type +definition must be omitted. + +Example: + +.. code:: ocaml + + (* Some third-party OCaml code *) + type message = { + from : string; + subject : string; + body : string; + } + +.. code:: ocaml + + (* + Our own ATD file used for making message_of_string and + string_of_message functions. + *) + type message <ocaml predef> = { + from : string; + subject : string; + body : string; + } + +Field ``mutable`` +~~~~~~~~~~~~~~~~~~~ + +Position: after a record field name + +Values: none, ``true`` or ``false`` + +Semantics: this flag indicates that the corresponding OCaml record field +is mutable. + +Example: + +.. code:: ocaml + + type counter = { + total <ocaml mutable> : int; + errors <ocaml mutable> : int; + } + +translates to the following OCaml definition: + +.. code:: ocaml + + type counter = { + mutable total : int; + mutable errors : int; + } + +Field ``default`` +~~~~~~~~~~~~~~~~~~~ + +Position: after a record field name marked with a ``\~{``} symbol or at +the beginning of a tuple field. + +Values: any valid OCaml expression + +Semantics: specifies an explicit default value for a field of an OCaml +record or tuple, allowing that field to be omitted. Default strings must +be escaped. + +Example: + +.. code:: ocaml + + type color = [ Black | White | Rgb of (int * int * int) ] + + type ford_t = { + year : int; + ~color <ocaml default="`Black"> : color; + ~name <ocaml default="\"Ford Model T\""> : string; + } + + type point = (int * int * <ocaml default="0"> : int) + +Field ``from`` +~~~~~~~~~~~~~~~~ + +Position: left-hand side of a type definition, after the type name + +Values: OCaml module name without the ``_t``, ``_b``, ``_j`` or ``_v`` +suffix. This can be also seen as the name of the original ATD file, +without the ``.atd`` extension and capitalized like an OCaml module +name. + +Semantics: specifies the base name of the OCaml modules where the type +and values coming with that type are defined. + +It is useful for ATD types defined as ``abstract`` and for types +annotated as predefined using the annotation ``<ocaml predef>``. In both +cases, the missing definitions must be provided by modules composed of +the base name and the standard suffix assumed by Atdgen which is ``_t``, +``_b``, ``_j`` or ``_v``. + +Example: First input file ``part1.atd``: + +.. code:: ocaml + + type point = { x : int; y : int } + +Second input file ``part2.atd`` depending on the first one: + +.. code:: ocaml + + type point <ocaml from="Part1"> = abstract + type points = point list + +To use a different type name than defined in the ``Part1`` module, add a +``t`` field declaration to the annotation which refers to the original +type name: + +.. code:: ocaml + + type point_xy <ocaml from="Part1" t="point"> = abstract + type points = point_xy list + + +Field ``module`` +~~~~~~~~~~~~~~~~~~ + +Using a custom wrapper +^^^^^^^^^^^^^^^^^^^^^^ + +Using the built-in ``wrap`` constructor, it is possible to add a layer +of abstraction on top of the concrete structure used for serialization. + +Position: after a ``wrap`` type constructor + +Values: OCaml module name + +A common use case is to parse strings used as unique identifiers and +wrap the result into an abstract type. Our OCaml module ``Uid`` needs to +provide a type ``t``, and two functions ``wrap`` and ``unwrap`` as +follows: + +.. code:: ocaml + + type t + val wrap : string -> t + val unwrap : t -> string + +Given that ``Uid`` OCaml module, we can write the following ATD +definition: + +.. code:: ocaml + + type uid = string wrap <ocaml module="Uid"> + +Other languages than OCaml using the same ATD type definitions may or +may not add their own abstract layer. Without an annotation, the +``wrap`` construct has no effect on the value being wrapped, i.e. +``wrap`` and ``unwrap`` default to the identity function. + +It is also possible to define ``t``, ``wrap``, and ``unwrap`` inline: + +.. code:: ocaml + + type uid = string wrap <ocaml t="Uid.t" + wrap="Uid.wrap" + unwrap="Uid.unwrap"> + +This can be useful for very simple validation: + +.. code:: ocaml + + type uid = string wrap + <ocaml wrap="fun s -> + if String.length s <> 16 then + failwith \"Invalid user ID\"; + s" + > + +Importing an external type definition +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In most cases since Atdgen 1.2.0 ``module`` annotations are deprecated +in favor of ``from`` annotations previously described. + +Position: left-hand side of a type definition, after the type name + +Values: OCaml module name + +Semantics: specifies the OCaml module where the type and values coming +with that type are defined. It is useful for ATD types defined as +``abstract`` and for types annotated as predefined using the annotation +``<ocaml predef>``. In both cases, the missing definitions can be +provided either by globally opening an OCaml module with an OCaml +directive or by specifying locally the name of the module to use. + +The latter approach is recommended because it allows to create type and +value aliases in the OCaml module being generated. It results in a +complete module signature regardless of the external nature of some +items. + +Example: Input file ``example.atd``: + +.. code:: ocaml + + type document <ocaml module="Doc"> = abstract + + type color <ocaml predef module="Color"> = + [ Black | White ] <ocaml repr="classic"> + + type point <ocaml predef module="Point"> = { + x : float; + y : float; + } + +gives the following OCaml type definitions (file ``example.mli``): + +.. code:: ocaml + + type document = Doc.document + + type color = Color.color = Black | White + + type point = Point.point = { x: float; y: float } + +Now for instance ``Example.Black`` and ``Color.Black`` can be used +interchangeably in other modules. + +Field ``t`` +~~~~~~~~~~~~~ + +Using a custom wrapper +^^^^^^^^^^^^^^^^^^^^^^ + +Specifies the OCaml type of an abstract ``wrap`` construct, possibly +overriding the default *M*\ ``.t`` if *M* is the module where the +``wrap`` and ``unwrap`` functions are found. + +Position: after a ``wrap`` type constructor + +Values: OCaml type name + +Example: + +.. code:: ocaml + + type uid = string wrap <ocaml module="Uid" t="Uid.uid"> + +is equivalent to: + +.. code:: ocaml + + type uid = string wrap <ocaml t="Uid.uid" wrap="Uid.wrap" unwrap="Uid.unwrap"> + +Importing an external type definition +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Position: left-hand side of a type definition, after the type name. Must +be used in conjunction with a ``module`` field. + +Values: OCaml type name as found in an external module. + +Semantics: This option allows to specify the name of an OCaml type +defined in an external module. + +It is useful when the type needs to be renamed because its original name +is already in use or not enough informative. Typically we may want to +give the name ``foo`` to a type originally defined in OCaml as +``Foo.t``. + +Example: + +.. code:: ocaml + + type foo <ocaml_biniou module="Foo" t="t"> = abstract + type bar <ocaml_biniou module="Bar" t="t"> = abstract + type t <ocaml_biniou module="Baz"> = abstract + +allows local type names to be unique and gives the following OCaml type +definitions: + +.. code:: ocaml + + type foo = Foo.t + type bar = Bar.t + type t = Baz.t + +Fields ``wrap`` and ``unwrap`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +See "Using a custom wrapper" under section ``ocaml``, fields +``module`` and ``t``. + +Field ``field_prefix`` +~~~~~~~~~~~~~~~~~~~~~~~~ + +Position: record type expression + +Values: any string making a valid prefix for OCaml record field names + +Semantics: specifies a prefix to be prepended to each field of the OCaml +definition of the record. Overridden by alternate field names defined on +a per-field basis. + +Example: + +.. code:: ocaml + + type point2 = { + x : int; + y : int; + } <ocaml field_prefix="p2_"> + +gives the following OCaml type definition: + +.. code:: ocaml + + type point2 = { + p2_x : int; + p2_y : int; + } + +Field ``name`` +~~~~~~~~~~~~~~~~ + +Position: after record field name or variant name + +Values: any string making a valid OCaml record field name or variant +name + +Semantics: specifies an alternate record field name or variant names to +be used in OCaml. + +Example: + +.. code:: ocaml + + type color = [ + Black <ocaml name="Grey0"> + | White <ocaml name="Grey100"> + | Grey <ocaml name="Grey50"> + ] + + type profile = { + id <ocaml name="profile_id"> : int; + username : string; + } + +gives the following OCaml type definitions: + +.. code:: ocaml + + type color = [ + `Grey0 + | `Grey100 + | `Grey50 + ] + + type profile = { + profile_id : int; + username : string; + } + +Field ``repr`` +~~~~~~~~~~~~~~~~ + +Integers +^^^^^^^^ + +Position: after ``int`` type + +Values: ``char``, ``int32``, ``int64``, ``float`` + +Semantics: specifies an alternate type for representing integers. The +default type is ``int``, but ``char``, ``int32``, ``int64`` or ``float`` +can be used instead. + +The three types ``char``, ``int32`` and ``int64`` are supported by both +Atdgen-biniou and Atdgen-json but Atdgen-biniou currently requires that +they map to the corresponding fixed-width types provided by the biniou +format. + +The type ``float`` is only supported in conjunction with JSON and is +useful when an OCaml float is used to represent an integral value, such +as a time in seconds returned by ``Unix.time()``. When converted into +JSON, floats are rounded to the nearest integer. + +Example: + +.. code:: ocaml + + type t = { + id : int + <ocaml repr="int64"> + <biniou repr="int64">; + data : string list; + } + +Lists and arrays +^^^^^^^^^^^^^^^^ + +Position: after a ``list`` type + +Values: ``array`` + +Semantics: maps to OCaml's ``array`` type instead of ``list``. + +Example: + +.. code:: ocaml + + type t = { + id : int; + data : string list + <ocaml repr="array">; + } + +Sum types +^^^^^^^^^ + +Position: after a sum type (denoted by square brackets) + +Values: ``classic`` + +Semantics: maps to OCaml's classic variants instead of polymorphic +variants. + +Example: + +.. code:: ocaml + + type fruit = [ Apple | Orange ] <ocaml repr="classic"> + +translates to the following OCaml type definition: + +.. code:: ocaml + + type fruit = Apple | Orange + +Shared values (obsolete) +^^^^^^^^^^^^^^^^^^^^^^^^ + +Position: after a ``shared`` type + +This feature is obsolete and was last supported by atdgen 1.3.1. + +Field ``valid`` +~~~~~~~~~~~~~~~~~ + +Since atdgen 1.6.0. + +Position: after any type expression except type variables + +Values: OCaml function that takes one argument of the given type and +returns a bool + +Semantics: ``atdgen -v`` produces for each type named *t* a function +``validate_``\ *t*: + +.. code:: ocaml + + val validate_t : Atdgen_runtime.Util.Validation.path -> t -> Atdgen_runtime.Util.Validation.error option + +Such a function returns ``None`` if and only if the value and all of its +subnodes pass all the validators specified by annotations of the form +``<ocaml validator="...">`` or ``<ocaml valid="...">`` (at most one per +node). + +Example: + +.. code:: ocaml + + type positive = int <ocaml validator="fun x -> x > 0"> + + type point = { + x : positive; + y : positive; + z : int; + } + <ocaml valid="Point.validate"> + (* Some validating function from a user-defined module Point *) + +The generated ``validate_point`` function calls the validator for the +containing object first (``Point.validate``) and continues on its fields +``x`` then ``y`` until an error is returned. + +.. code:: ocaml + + match validate_point [] { x = 1; y = 0; z = 1 } with + | None -> () + | Some e -> + Printf.eprintf "Error: %s\n%!" + (Atdgen_runtime.Util.Validation.string_of_error e) + +The above code prints the following error message: + +:: + + Error: Validation error; path = <root>.y + +In order to customize the error message and print the faulty value, use +``validator`` instead of ``valid``, as described next. + +Field ``validator`` +~~~~~~~~~~~~~~~~~~~~~ + +This is a variant of the ``valid`` annotation that allows full control +over the error message that gets generated in case of an error. + +Position: after any type expression except type variables + +Values: OCaml function that takes the path in current JSON structure and +the object to validate, and returns an optional error. + +Semantics: ``atdgen -v`` produces for each type named *t* a function +``validate_``\ *t*: + +.. code:: ocaml + + val validate_t : Atdgen_runtime.Util.Validation.path -> t -> Atdgen_runtime.Util.Validation.error option + +Such a function returns ``None`` if and only if the value and all of its +subnodes pass all the validators specified by annotations of the form +``<ocaml validator="...">`` or ``<ocaml valid="...">`` (at most one per +node). + +Example: + +.. code:: ocaml + + type positive = int <ocaml validator=" + fun path x -> + if x > 0 then None + else + Some ( + Atdgen_runtime.Util.Validation.error + ~msg: (\"Not a positive integer: \" ^ string_of_int x) + path + ) + "> + + type point = { + x : positive; + y : positive; + z : int; + } + <ocaml validator="Point.validate"> + (* Some validating function from a user-defined module Point *) + +The following user code + +.. code:: ocaml + + match Toto_v.validate_point [] { x = 1; y = 0; z = 1 } with + | None -> () + | Some e -> + Printf.eprintf "Error: %s\n%!" + (Atdgen_runtime.Util.Validation.string_of_error e) + +results in printing: + +:: + + Error: Validation error: Not a positive integer: 0; path = <root>.y + +Section ``ocaml_biniou`` +-------------------------- + +Section ``ocaml_biniou`` takes precedence over section ``ocaml`` in +Biniou mode (``-b``) for the following fields: + +- ``predef`` (see section ``ocaml``, field ``predef``) +- ``module`` (see section ``ocaml``, field ``module``) +- ``t`` (see section ``ocaml.t``) + +Section ``ocaml_json`` (obsolete) +----------------------------------- + +Section ``ocaml_json`` takes precedence over section ``ocaml`` in JSON +mode (``-json`` or ``-j``) for the following fields: + +- ``predef`` (see section ``ocaml``, field ``predef``) +- ``module`` (see section ``ocaml``, field ``module``) +- ``t`` (see section ``ocaml``, field ``t``) + +Please note that ``atdgen -json`` is now deprecated in favor of +``atdgen -j`` (json) and ``atdgen -t`` (types). The latter is in charge +of producing type definitions independently from JSON and will ignore +``<ocaml_json ...>`` annotations, making them almost useless. The +equivalent ``<ocaml ...>`` annotations are almost always preferable. + +Example: + +This example shows how to parse a field into a generic tree of type +``Yojson.Safe.t`` rather than a value of a specialized OCaml type. + +.. code:: ocaml + + type dyn <ocaml_json module="Yojson.Safe" t="json"> = abstract + + type t = { foo: int; bar: dyn } + +translates to the following OCaml type definitions: + +.. code:: ocaml + + type dyn = Yojson.Safe.t + + type t = { foo : int; bar : dyn } + +Sample OCaml value of type ``t``: + +.. code:: ocaml + + { + foo = 12345; + bar = + `List [ + `Int 12; + `String "abc"; + `Assoc [ + "x", `Float 3.14; + "y", `Float 0.0; + "color", `List [ `Float 0.3; `Float 0.0; `Float 1.0 ] + ] + ] + } + +Corresponding JSON data as obtained with ``string_of_t``: + +:: + + {"foo":12345,"bar":[12,"abc",{"x":3.14,"y":0.0,"color":[0.3,0.0,1.0]}]} + +Section ``doc`` +----------------- + +Unlike comments, ``doc`` annotations are meant to be propagated into the +generated source code. This is useful for making generated interface +files readable without having to consult the original ATD file. + +Generated source code comments can comply to a standard format and take +advantage of documentation generators such as javadoc or ocamldoc. + +Field ``text`` +~~~~~~~~~~~~~~~~ + +Position: + +- after the type name on the left-hand side of a type definition +- after the type expression on the right hand of a type definition (but + not after any type expression) +- after record field names +- after variant names + +Values: UTF-8-encoded text using a minimalistic markup language + +Semantics: The markup language is defined as follows: + +- Blank lines separate paragraphs. +- ``{{ }}`` can be used to enclose inline verbatim text. +- ``{{{ }}}`` can be used to enclose verbatim text where whitespace is + preserved. +- The backslash character is used to escape special character + sequences. In regular paragraph mode the special sequences are ``\``, + ``{{`` and ``{{{``. In inline verbatim text, special sequences are + ``\`` and ``}}``. In verbatim text, special sequences are ``\`` and + ``}}}``. + +Example: The following is an example demonstrating the use of ``doc`` +annotations generated using: + +:: + + $ atdgen -t ocamldoc_example.atd + +Input file ``ocamldoc_example.atd``: + +.. code:: ocaml + + <doc text="This is the title"> + + type point = { + x <doc text="The first coordinate">: float; + y <doc text="The second coordinate">: float; + } + <doc text=" + The type of a point. A value {{p}} can be created as follows: + {{{ + let p = { x = 1.2; y = 5.0 } + }}} + "> + + type color = [ + | Black <doc text="Same as {{RGB (0,0,0)}}"> + | White <doc text="Same as {{RGB (255, 255, 255)}}"> + | RGB + <doc text="Red, green, blue components"> + of (int * int * int) + ] + +translates using ``atdgen -t ocamldoc_example.atd`` into the following +OCaml interface file ``ocamldoc_example_t.mli`` with ocamldoc-compliant +comments: + +.. code:: ocaml + + (* Auto-generated from "ocamldoc_example.atd" *) + + + (** This is the title *) + + (** + The type of a point. A value [p] can be created as follows: + + {v + let p = \{ x = 1.2; y = 5.0 \} + v} + *) + type point = { + x: float (** The first coordinate *); + y: float (** The second coordinate *) + } + + type color = [ + `Black (** Same as [RGB (0,0,0)] *) + | `White (** Same as [RGB (255, 255, 255)] *) + | `RGB of (int * int * int) (** Red, green, blue components *) + ] + +Atdgen runtime library +====================== + +A library named `atdgen-runtime <https://github.com/mjambon/atd/tree/master/atdgen-runtime/src>`_ is installed by the standard installation +process. Only a fraction of it is officially supported and documented. + +Modules intended for all users are: + +- ``Util`` +- ``Json_adapter`` + +The other modules exported by the library are used directly by +generated code. Tool developers may use them but we don't guarantee +strong compatibility across releases. diff --git a/doc/atdj.rst b/doc/atdj.rst new file mode 100644 index 0000000..adbd37b --- /dev/null +++ b/doc/atdj.rst @@ -0,0 +1,319 @@ +*************************** +Java/Json support with atdj +*************************** + +The ATDJ tool generates a Java interface from an ATD interface. In +particular, given a set of ATD types, this tool generates a set of Java +classes representing those types. These classes may then be instantiated +from JSON representations of those same ATD types. + +The primary benefits of using the generated interface, over manually +manipulating JSON strings from within Java, are safety and ease of use. +Specifically, the generated interface offers the following features: + +- JSON strings are automatically checked for correctness with respect + to the ATD specificion. + +- Details such as optional fields and their associated default values + are automatically handled. + +- Several utility methods are included “for free”. These support + equality testing, the visitor pattern and conversion back to JSON. + +Installation +============ + +Build and install the ``atdj`` command with `opam <https://opam.ocaml.org/>`__: + + :: + + opam install atdj + +Quick-start +=========== + +In this section we briefly describe how to to generate a Java interface +from an example ATD file ``test.atd``. We then show how to build and run +an example application ``AtdjTest`` that uses the generated interface. + +#. Generate and compile the interface: + + :: + + atdj -graph -package com.mylife.test test.atd + export CLASSPATH='.:json.jar' + javac com/mylife/test/*.java + +#. Compile and run the example, saving the output for later inspection: + + :: + + javac AtdjTest.java + java AtdjTest >test.out + +#. Optionally, generate Javadoc documentation: + + :: + + javadoc -d doc -public com.mylife.test + + The resulting documentation is located in the directory ``doc``. + +#. Optionally, generate a class graph of the generated interface: + + :: + + dot -Tpdf test.dot >test.pdf + +The output file ``test.pdf`` contains a class graph of the generated +Java interface. The required ``dot`` program is part of the Graphviz +graph visualisation package, and may be downloaded from +http://www.graphviz.org/. + +In the following sections we discuss the individual steps in more +detail, using the example from above. + +Generating the interface +======================== + +In this section we describe the process of generating a Java interface +from an ATD specification. + +A Java interface is generated from an ATD file as + +:: + + atdj -package <package> <atd_file> + +This outputs a set of Java source files. The ``-package`` option causes +the resulting classes to be members of the specified package, and also +to be located in the corresponding output directory. If no package is +specified, then the default package of ``out`` is used. + +For example, the command + +:: + + atdj -graph -package com.mylife.test test.atd + +causes the generated files to be members of the package +``com.mylife.test`` and to be located in the directory +``com/mylife/test``. + +The generated source files reference various members of the included +org.json package. Therefore, in order to compile the generated files, +the ``org.json`` package must be located within the Java classpath. +Supposing that the ``org.json`` package is located within the archive +``json.jar`` within the current directory, it is sufficient to set the +classpath as follows: + +:: + + export CLASSPATH='json.jar' + +Returning to our example, the generated source files may then be +compiled as: + +:: + + javac com/mylife/test/*.java + +Generating Javadoc documentation +================================ + +The generated Java code contains embedded Javadoc comments. These may be +extracted to produce Javadoc documentation. In the case of our example, +it is sufficient to run the following command: + +:: + + javadoc -d doc/example -public com.mylife.test + +Generating a class graph +======================== + +We now discuss the ``-graph`` option of ATDJ. When enabled, this causes +ATDJ to output a graph of the class hierarchy of the generated code. The +output is intended to document the generated code, helping users to +avoid consulting the source code. + +Continuing with our example, the use of this option results in the +generation of an additional output file named ``test.dot``. Assuming +that the ``dot`` program is installed, a PDF class graph named +``test.pdf`` can then created by running the command + +:: + + dot -Tpdf test.dot >test.pdf + +In the generated class graph, rectangular and oval nodes correspond to +classes and interfaces, respectively. Field names are specified in the +second line of retangular (class) nodes. Solid arcs denote subtyping +(``implements``/``extends``), whilst dashed arcs link fields to their +types. + +Translation reference +===================== + +In this section we informally define how Java types are generated from +ATD types. + +Bools, ints, floats, string, lists +---------------------------------- + ++---------------+------------------+ +| ATD type, t | Java type, <t> | ++===============+==================+ +| bool | boolean | ++---------------+------------------+ +| int | int | ++---------------+------------------+ +| float | double | ++---------------+------------------+ +| string | String | ++---------------+------------------+ +| t list | <t>[] | ++---------------+------------------+ + +Options +------- + +Suppose that we have ATD type ``t option``. Then this is translated into +the following Java reference type: + +:: + + public class CNAME implements Atdj { + // Constructor + public CNAME(String s) throws JSONException { ... } + + // Get the optional value, if present + public CNAME get() throws JSONException { ... } + + // Comparison and equality + public int compareTo(CNAME that) { ... } + public boolean equals(CNAME that) { ... } + + public <t> value; // The value + public boolean is_set; // Whether the value is set + } + +Records +------- + +Suppose that we have the ATD record type + +:: + + { f_1: t_1 + ; ... + ; f_n: t_n + } + +Then this is translated into the following Java reference type: + +:: + + public class CNAME implements Atdj { + // Constructor + public CNAME(String s) throws JSONException { ... } + + // Comparison and equality + public int compareTo(CNAME that) { ... } + public boolean equals(CNAME that) { ... } + + // The individual fields + public <t_1> f_1; + ... + public <t_n> f_n; + } + +An optional field ``~f_i: t_i`` causes the class field ``f_i`` to be +given a default value of type ``<t_i>`` if the field is absent from the +JSON string used to instantiate the class. The default values are as +follows: + ++------------+---------------------------------------+ +| ATD type | Default Java value | ++============+=======================================+ +| bool | false | ++------------+---------------------------------------+ +| int | 0 | ++------------+---------------------------------------+ +| float | 0.0 | ++------------+---------------------------------------+ +| string | “” | ++------------+---------------------------------------+ +| t list | Empty array | ++------------+---------------------------------------+ +| t option | Optional value with is\_set = false | ++------------+---------------------------------------+ + +Default values cannot be defined for record and sum types. + +An optional field ``?f_i: t_i option`` has the same default behaviour as +above, with the additional behaviour that if the field is present in the +JSON string then the value must be of type <t> (not <t> option); the +value is then automatically lifted into a <t> option, with is\_set = +true. + +Sums +---- + +Suppose that we have the ATD sum type + +:: + + [ C_1 of t_1 + | ... + | C_n of t_n + ] + +Then this is translated into the following Java reference types: + +:: + + public interface IFCNAME extends Atdj { + public int compareTo(IFCNAME that); + public boolean equals(IFCNAME that); + ... + } + +:: + + public class CNAME_i implements IFCNAME, Atdj { + // Comparison and equality + public int compareTo(CNAME that) { ... } + public boolean equals(CNAME that) { ... } + + public <t_i> value; + } + +The value field is absent if the constructor C\_i has no argument. + +The Atdj and Visitor interfaces +------------------------------- + +All generated reference types additionally implement the interface + +:: + + interface Atdj { + String toString(); + String toString(int indent); + int hashCode(); + Visitor accept(Visitor v); + } + +where the Visitor interface is defined as + +:: + + public interface Visitor { + public void visit(CNAME_1 value); + ... + public void visit(CNAME_n value); + } + +for generated reference types ``CNAME``\ \_i. Visit methods for +primitive and optional primitive types are omitted. diff --git a/doc/biniou-format.txt b/doc/biniou-format.txt new file mode 100644 index 0000000..0a5445c --- /dev/null +++ b/doc/biniou-format.txt @@ -0,0 +1,176 @@ + The Biniou format + ----------------- + +Contents: + +1. Grammar +2. Tags +3. Fixed-length types +4. Vints +5. Field and variant name hashing +6. Numeric variants + + + +1. Grammar + + + TAGVAL ::= TAG VAL // A biniou value with its matching tag + + VAL ::= ATOM + | ARRAY + | TUPLE + | RECORD + | NUM_VARIANT + | VARIANT + | TABLE + | SHARED + + ATOM ::= unit // 0, using one byte + | bool // 0 for false, 1 for true, using one byte + | int8 // 1 arbitrary byte + | int16 // 2 arbitrary bytes + | int32 // 4 arbitrary bytes + | int64 // 8 arbitrary bytes + | float64 // IEEE-754 binary64 + | uvint // unsigned variable-length int + | svint // signed variable-length int + | string // sequence of any number of bytes + + ARRAY ::= LENGTH (TAG VAL* )? + NUM_VARIANT ::= NUM_VARIANT_TAG TAGVAL? + VARIANT ::= VARIANT_TAG TAGVAL? + TUPLE ::= LENGTH TAGVAL* + RECORD ::= LENGTH (FIELD_TAG TAGVAL)* + TABLE ::= LENGTH (LENGTH (FIELD_TAG TAG)* (VAL* )* )? // list of records + + SHARED ::= OFFSET TAGVAL? // Value given iff the offset is 0. + // Otherwise, the offset indicates the + // relative position to the left of a SHARED + // to which we are redirected. + + TAG ::= int8 // identifies a type of node + LENGTH ::= uvint + OFFSET ::= uvint + NUM_VARIANT_TAG ::= int8 // 0-127 if no argument, 128-255 if has argument + VARIANT_TAG ::= int32 // first bit indicates argument, then 31-bit hash + FIELD_TAG ::= int32 // 31-bit hash (first bit always 1) + + + +2. Tags + + +Tags indicate the shallow structure of any biniou value. + +The biniou format is such that the tag of any value is known +from the input data. This allows decoding biniou data as a tree +where each node represents a biniou value, without requiring external +type information. + +The tag values for the various kinds of biniou values are: + + Type of value Tag + --------------------------- + bool 0 + int8 1 + int16 2 + int32 3 + int64 4 + float64 12 + uvint 16 + svint 17 + string 18 + ARRAY 19 + TUPLE 20 + RECORD 21 + NUM_VARIANT 22 + VARIANT 23 + unit 24 + TABLE 25 + SHARED 26 + + + +3. Fixed-length types + + +Atomic values of type unit, bool, int8, int16, int32, int64 and float64 +represent arbitrary sequences of 1, 2, 4 or 8 bytes. + +In order to make the visualization of data easier, +the default interpretation of these values shall be used: + + Length + in bytes Type of value Default interpretation + --------------------------------------------------------------------- + 1 unit 0 represents the unit value + 1 bool 0 represents false, 1 represents true + 1 int8 unsigned 8-bit int + 2 int16 big endian unsigned 16-bit int + 4 int32 big endian unsigned 32-bit int + 8 int64 big endian unsigned 64-bit int + 8 float64 big endian IEEE-754 binary64 (double) + + + +4. Vints + + +Vints are a variable-length, byte-aligned representation of +positive integers. + +A vint is represented by a sequence of bytes from least significant +to most significant. In all the bytes except the last one, the +high bit is set to 1 and indicates that more bytes follow. +The high bit of the last byte is set to 0. +The remaining 7 bits in each byte represent data. + +Here is the representation of some sample values: + + 0xxxxxxx + 0 00000000 + 1 00000001 + 2 00000010 + 127 01111111 + + 1xxxxxxx 0xxxxxxx + 128 10000000 00000001 + 129 10000001 00000001 + 255 11111111 00000001 + 256 11111111 00000010 + 16383 11111111 01111111 + + 1xxxxxxx 1xxxxxxx 0xxxxxxx + 16384 10000000 10000000 00000001 + 16385 10000001 10000000 00000001 + + +Positive integers can be represented by standard vints. +We call this representation unsigned vint or uvint. + +Arbitrary integers can also be represented using vints, after mapping +to positive integers. We call this representation signed vint or svint. +Positive numbers and 0 are mapped to even numbers and negative numbers +are mapped to odd positive numbers. Here is the mapping for +small numbers: + + vint unsigned signed + representation interpretation interpretation + (uvint) (svint) + 0xxxxxx0 + 00000000 0 0 + 00000010 2 1 + 00000100 4 2 + 00000110 6 3 + + 0xxxxxx1 + 00000001 1 -1 + 00000011 3 -2 + 00000101 5 -3 + + + +5. Field and variant name hashing + + diff --git a/doc/conf.py b/doc/conf.py index 73562a3..cb497d9 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -44,8 +44,8 @@ master_doc = 'index' # General information about the project. project = 'atd' -copyright = u'2017, Martin Jambon' -author = u'Martin Jambon' +copyright = u'2010-2012 MyLife; 2012-2018 Martin Jambon & contributors' +author = u'Martin Jambon, John Billings & contributors' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/doc/index.rst b/doc/index.rst index 9260ea5..d597be2 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -1,4 +1,4 @@ -Welcome to atd's documentation! +Welcome to ATD's documentation! =============================== .. toctree:: @@ -6,3 +6,5 @@ Welcome to atd's documentation! tutorial syntax + atdgen + atdj diff --git a/doc/syntax.rst b/doc/syntax.rst index 15fe690..b035bff 100644 --- a/doc/syntax.rst +++ b/doc/syntax.rst @@ -1,22 +1,6 @@ -**************** -Syntax Reference -**************** - -Name -==== - -atd - syntax for cross-language **a**djustable **t**ype **d**efinitions - -Synopsis -======== - -:: - - atdcat [_infile_.atd] [options...] - - atdcat -version - - atdcat -help +************************* +ATD core syntax reference +************************* Introduction ============ @@ -511,15 +495,15 @@ any monomorphic type expression. It allows notably to represent cyclic values and to enforce that cycles are preserved during transformations such as serialization. -```ocaml -(* Example of a simple graph type *) -type shared_node = node shared (* sharing point *) -type graph = shared_node list -type node = { - label : string; - neighbors : shared_node list; -} -``` +.. code-block:: ocaml + + (* Example of a simple graph type *) + type shared_node = node shared (* sharing point *) + type graph = shared_node list + type node = { + label : string; + neighbors : shared_node list; + } Two shared values that are physically identical must remain physically identical after any translation from one data format to another. diff --git a/doc/tutorial-data/Makefile b/doc/tutorial-data/Makefile new file mode 100644 index 0000000..3a1e722 --- /dev/null +++ b/doc/tutorial-data/Makefile @@ -0,0 +1,4 @@ +.PHONY: clean +clean: + rm -f */*~ + rm -f */*.cm[ioxa] */*.cmx[as] */*.[oa] */*_[tjv].mli */*_[tjv].ml diff --git a/doc/tutorial-data/config-file/bad-config1.json b/doc/tutorial-data/config-file/bad-config1.json new file mode 100644 index 0000000..258e392 --- /dev/null +++ b/doc/tutorial-data/config-file/bad-config1.json @@ -0,0 +1,13 @@ +{ + "title": "Example", + "credentials": [ + { + "name": 0, + "key": "db7c0877bdef3016" + }, + { + "name": "tester", + "key": "09871ff387ac2b10" + } + ] +} diff --git a/doc/tutorial-data/config-file/bad-config2.json b/doc/tutorial-data/config-file/bad-config2.json new file mode 100644 index 0000000..5bab218 --- /dev/null +++ b/doc/tutorial-data/config-file/bad-config2.json @@ -0,0 +1,14 @@ +{ + "title": "Example", + "tiemout": 20, + "credentials": [ + { + "name": "joeuser", + "key": "db7c0877bdef3016" + }, + { + "name": "tester", + "key": "09871ff387ac2b10" + } + ] +} diff --git a/doc/tutorial-data/config-file/config.atd b/doc/tutorial-data/config-file/config.atd new file mode 100644 index 0000000..182dd8a --- /dev/null +++ b/doc/tutorial-data/config-file/config.atd @@ -0,0 +1,15 @@ +type config = { + title : string; + ?description : string option; + ~timeout <ocaml default="10"> : int; + ~credentials : param list + <ocaml valid="fun l -> + l <> [] || failwith \"missing credentials\"">; +} + +type param = { + name : string + <ocaml valid="fun s -> s <> \"\"">; + key : string + <ocaml valid="fun s -> String.length s = 16">; +} diff --git a/doc/tutorial-data/config-file/config.ml b/doc/tutorial-data/config-file/config.ml new file mode 100644 index 0000000..8881e72 --- /dev/null +++ b/doc/tutorial-data/config-file/config.ml @@ -0,0 +1,93 @@ +open Printf + +let param_template = + (* Sample item used to populate the template config file *) + { + Config_v.name = "foo"; + key = "0123456789abcdef" + } + +let config_template = + (* + Records can be conveniently created using functions generated by + "atdgen -v". + Here we use Config_v.create_config to create a record of type + Config_t.config. The big advantage over creating the record + directly using the record notation {...} is that we don't have to + specify default values (such as timeout in this example). + *) + Config_v.create_config ~title:"" ~credentials: [param_template] () + +let make_json_template () = + (* Thanks to the -j-defaults flag passed to atdgen, even default + fields will be printed out *) + let compact_json = Config_j.string_of_config config_template in + Yojson.Safe.prettify compact_json + +let print_template () = + print_endline (make_json_template ()) + +let print_format () = + print_string Config_atd.contents + +let validate fname = + let x = + try + (* Read config data structure from JSON file *) + let x = Atdgen_runtime.Util.Json.from_file Config_j.read_config fname in + (* Call the validators specified by <ocaml valid=...> *) + if not (Config_v.validate_config x) then + failwith "Some fields are invalid" + else + x + with e -> + (* Print decent error message and exit *) + let msg = + match e with + Failure s + | Yojson.Json_error s -> s + | e -> Printexc.to_string e + in + eprintf "Error: %s\n%!" msg; + exit 1 + in + (* Convert config to compact JSON and pretty-print it. + ~std:true means that the output will not use extended syntax for + variants and tuples but only standard JSON. *) + let json = Yojson.Safe.prettify ~std:true (Config_j.string_of_config x) in + print_endline json + +type action = Template | Format | Validate of string + +let main () = + let action = ref Template in + let options = [ + "-template", Arg.Unit (fun () -> action := Template), + " + prints a sample configuration file"; + + "-format", Arg.Unit (fun () -> action := Format), + " + prints the format specification of the config files (atd format)"; + + "-validate", Arg.String (fun s -> action := Validate s), + "<CONFIG FILE> + reads a config file, validates it, adds default values + and prints the config nicely to stdout"; + ] + in + let usage_msg = sprintf "\ +Usage: %s [-template|-format|-validate ...] +Demonstration of how to manage JSON configuration files with atdgen. +" + Sys.argv.(0) + in + let anon_fun s = eprintf "Invalid command parameter %S\n%!" s; exit 1 in + Arg.parse options anon_fun usage_msg; + + match !action with + Template -> print_template () + | Format -> print_format () + | Validate s -> validate s + +let () = main () diff --git a/doc/tutorial-data/config-file/demo.sh b/doc/tutorial-data/config-file/demo.sh new file mode 100755 index 0000000..078e959 --- /dev/null +++ b/doc/tutorial-data/config-file/demo.sh @@ -0,0 +1,37 @@ +#! /bin/sh -e + +set -x + +# Embed the contents of the .atd file into our OCaml program +echo 'let contents = "\' > config_atd.ml +sed -e 's/\([\\"]\)/\\\1/g' config.atd >> config_atd.ml +echo '"' >> config_atd.ml + +# Derive OCaml type definitions from .atd file +atdgen -t config.atd + +# Derive JSON-related functions from .atd file +atdgen -j -j-defaults -j-strict-fields config.atd + +# Derive validator from .atd file +atdgen -v config.atd + +# Compile the OCaml program +ocamlfind ocamlopt -o config \ + config_t.mli config_t.ml config_j.mli config_j.ml config_v.mli config_v.ml \ + config_atd.ml config.ml -package atdgen -linkpkg + +# Output a sample config +./config -template + +# Print the original type definitions +./config -format + +# Fail to validate an invalid config file +./config -validate bad-config1.json || : + +# Fail to validate another invalid config file (using custom validators) +./config -validate bad-config3.json || : + +# Validate, inject missing defaults and pretty-print +./config -validate sample-config.json diff --git a/doc/tutorial-data/config-file/sample-config.json b/doc/tutorial-data/config-file/sample-config.json new file mode 100644 index 0000000..4a1ac2b --- /dev/null +++ b/doc/tutorial-data/config-file/sample-config.json @@ -0,0 +1,13 @@ +{ + "title": "Example", + "credentials": [ + { + "name": "joeuser", + "key": "db7c0877bdef3016" + }, + { + "name": "tester", + "key": "09871ff387ac2b10" + } + ] +} diff --git a/doc/tutorial-data/hello/demo.sh b/doc/tutorial-data/hello/demo.sh new file mode 100755 index 0000000..81950c4 --- /dev/null +++ b/doc/tutorial-data/hello/demo.sh @@ -0,0 +1,15 @@ +#! /bin/sh -e + +set -x +cat hello.atd +atdgen -t hello.atd +atdgen -j hello.atd +ls +ocamlfind ocamlc -c hello_t.mli -package atdgen +ocamlfind ocamlc -c hello_j.mli -package atdgen +ocamlfind ocamlopt -c hello_t.ml -package atdgen +ocamlfind ocamlopt -c hello_j.ml -package atdgen +ocamlfind ocamlopt -c hello.ml -package atdgen +ocamlfind ocamlopt -o hello hello_t.cmx hello_j.cmx hello.cmx \ + -package atdgen -linkpkg +./hello diff --git a/doc/tutorial-data/hello/hello.atd b/doc/tutorial-data/hello/hello.atd new file mode 100644 index 0000000..3e3b6d1 --- /dev/null +++ b/doc/tutorial-data/hello/hello.atd @@ -0,0 +1,5 @@ +type date = { + year : int; + month : int; + day : int; +} diff --git a/doc/tutorial-data/hello/hello.ml b/doc/tutorial-data/hello/hello.ml new file mode 100644 index 0000000..41e14bc --- /dev/null +++ b/doc/tutorial-data/hello/hello.ml @@ -0,0 +1,4 @@ +open Hello_t +let () = + let date = { year = 1970; month = 1; day = 1 } in + print_endline (Hello_j.string_of_date date) diff --git a/doc/tutorial-data/inspect-biniou/demo.sh b/doc/tutorial-data/inspect-biniou/demo.sh new file mode 100755 index 0000000..b41950e --- /dev/null +++ b/doc/tutorial-data/inspect-biniou/demo.sh @@ -0,0 +1,18 @@ +#! /bin/sh -e + +set -x + +cat tree.atd +cat tree.ml + +atdgen -t tree.atd +atdgen -b tree.atd +ocamlfind ocamlopt -o tree \ + tree_t.mli tree_t.ml tree_b.mli tree_b.ml tree.ml \ + -package atdgen -linkpkg +./tree + +ls -l tree.dat +bdump tree.dat +bdump -w Empty,Node tree.dat +bdump tree.dat diff --git a/doc/tutorial-data/inspect-biniou/tree.atd b/doc/tutorial-data/inspect-biniou/tree.atd new file mode 100644 index 0000000..de356e1 --- /dev/null +++ b/doc/tutorial-data/inspect-biniou/tree.atd @@ -0,0 +1,5 @@ +(* This a binary tree. Just for the purpose of pretty-printing trees. *) +type tree = + [ Empty + | Node of (tree * int * tree) ] + diff --git a/doc/tutorial-data/inspect-biniou/tree.ml b/doc/tutorial-data/inspect-biniou/tree.ml new file mode 100644 index 0000000..c08d70b --- /dev/null +++ b/doc/tutorial-data/inspect-biniou/tree.ml @@ -0,0 +1,30 @@ +open Printf + +(* sample value *) +let tree : Tree_t.tree = + `Node ( + `Node (`Empty, 1, `Empty), + 2, + `Node ( + `Node (`Empty, 3, `Empty), + 4, + `Node (`Empty, 5, `Empty) + ) + ) + +let () = + (* write sample value to file *) + let fname = "tree.dat" in + Atdgen_runtime.Util.Biniou.to_file Tree_b.write_tree fname tree; + + (* write sample value to string *) + let s = Tree_b.string_of_tree tree in + printf "raw value (saved as %s):\n%S\n" fname s; + printf "length: %i\n" (String.length s); + + printf "pretty-printed value (without dictionary):\n"; + print_endline (Bi_io.view s); + + printf "pretty-printed value (with dictionary):\n"; + let unhash = Bi_io.make_unhash ["Empty"; "Node"; "foo"; "bar" ] in + print_endline (Bi_io.view ~unhash s) diff --git a/doc/tutorial-data/modularity/demo.sh b/doc/tutorial-data/modularity/demo.sh new file mode 100755 index 0000000..3c42b51 --- /dev/null +++ b/doc/tutorial-data/modularity/demo.sh @@ -0,0 +1,23 @@ +#! /bin/sh -e + +set -x +cat part1.atd +cat part2.atd +cat part3.atd +for x in part1 part2 part3; do + atdgen -t $x.atd + atdgen -j $x.atd + ocamlfind ocamlc -c ${x}_t.mli -package atdgen + ocamlfind ocamlc -c ${x}_j.mli -package atdgen + ocamlfind ocamlopt -c ${x}_t.ml -package atdgen + ocamlfind ocamlopt -c ${x}_j.ml -package atdgen +done +ocamlfind ocamlopt -c main.ml -package atdgen + +ocamlfind ocamlopt -o test_modularity \ + part1_t.cmx part1_j.cmx \ + part2_t.cmx part2_j.cmx \ + part3_t.cmx part3_j.cmx \ + main.cmx \ + -package atdgen -linkpkg +./test_modularity diff --git a/doc/tutorial-data/modularity/main.ml b/doc/tutorial-data/modularity/main.ml new file mode 100644 index 0000000..c799ba8 --- /dev/null +++ b/doc/tutorial-data/modularity/main.ml @@ -0,0 +1,11 @@ +let v = { + Part3_t.name = "foo"; + data = Some [ + { Part1_t.x = 1; y = 2 }; + { Part1_t.x = 3; y = 4 }; + ] +} + +let () = + Atdgen_runtime.Util.Json.to_channel Part3_j.write_t3 stdout v; + print_newline () diff --git a/doc/tutorial-data/modularity/part1.atd b/doc/tutorial-data/modularity/part1.atd new file mode 100644 index 0000000..ada1b7c --- /dev/null +++ b/doc/tutorial-data/modularity/part1.atd @@ -0,0 +1 @@ +type t = { x : int; y : int } diff --git a/doc/tutorial-data/modularity/part2.atd b/doc/tutorial-data/modularity/part2.atd new file mode 100644 index 0000000..53e2dbc --- /dev/null +++ b/doc/tutorial-data/modularity/part2.atd @@ -0,0 +1,8 @@ +type t1 <ocaml from="Part1" t="t"> = abstract + (* + Imports type t defined in file part1.atd. + The local name is t1. Because the local name (t1) is different from the + original name (t), we must specify the original name using t=. + *) + +type t2 = t1 list diff --git a/doc/tutorial-data/modularity/part3.atd b/doc/tutorial-data/modularity/part3.atd new file mode 100644 index 0000000..d8c62e6 --- /dev/null +++ b/doc/tutorial-data/modularity/part3.atd @@ -0,0 +1,6 @@ +type t2 <ocaml from="Part2"> = abstract + +type t3 = { + name : string; + ?data : t2 option; +} diff --git a/doc/tutorial-data/pretty-json/demo.sh b/doc/tutorial-data/pretty-json/demo.sh new file mode 100755 index 0000000..1fa4b24 --- /dev/null +++ b/doc/tutorial-data/pretty-json/demo.sh @@ -0,0 +1,11 @@ +#! /bin/sh -e + +set -x +cat single.json +ydump single.json +cat stream.json +ydump -s stream.json + +cat prettify.ml +ocamlfind ocamlopt -o prettify prettify.ml -package atdgen -linkpkg +./prettify diff --git a/doc/tutorial-data/pretty-json/prettify.ml b/doc/tutorial-data/pretty-json/prettify.ml new file mode 100644 index 0000000..1ce5f63 --- /dev/null +++ b/doc/tutorial-data/pretty-json/prettify.ml @@ -0,0 +1,5 @@ +let json = +"[1234,\"abcde\",{\"start_date\":{\"year\":1970,\"month\":1,\"day\":1}, +\"end_date\":{\"year\":1980,\"month\":1,\"day\":1}}]" + +let () = print_endline (Yojson.Safe.prettify json) diff --git a/doc/tutorial-data/pretty-json/single.json b/doc/tutorial-data/pretty-json/single.json new file mode 100644 index 0000000..540d7c3 --- /dev/null +++ b/doc/tutorial-data/pretty-json/single.json @@ -0,0 +1,2 @@ +[1234,"abcde",{"start_date":{"year":1970,"month":1,"day":1}, +"end_date":{"year":1980,"month":1,"day":1}}] diff --git a/doc/tutorial-data/pretty-json/stream.json b/doc/tutorial-data/pretty-json/stream.json new file mode 100644 index 0000000..1b2a785 --- /dev/null +++ b/doc/tutorial-data/pretty-json/stream.json @@ -0,0 +1,3 @@ +[1234,"abcde",{"start_date":{"year":1970,"month":1,"day":1}, +"end_date":{"year":1980,"month":1,"day":1}}] +[1,"a",{}] diff --git a/doc/tutorial-data/untypable-json/input.json b/doc/tutorial-data/untypable-json/input.json new file mode 100644 index 0000000..de29253 --- /dev/null +++ b/doc/tutorial-data/untypable-json/input.json @@ -0,0 +1,20 @@ +[ + { + "label": "flower", + "value": { + "petals": [12, 45, 83.5555], + "water": "a340bcf02e" + } + }, + { + "label": "flower", + "value": { + "petals": "undefined", + "fold": null, + "water": 0 + } + }, + { "labels": ["fork", "scissors"], + "value": [ 8, 8 ] + } +] diff --git a/doc/tutorial-data/untypable-json/untypable.atd b/doc/tutorial-data/untypable-json/untypable.atd new file mode 120000 index 0000000..1af3ae2 --- /dev/null +++ b/doc/tutorial-data/untypable-json/untypable.atd @@ -0,0 +1 @@ +untypable_v2.atd
\ No newline at end of file diff --git a/doc/tutorial-data/untypable-json/untypable_v1.atd b/doc/tutorial-data/untypable-json/untypable_v1.atd new file mode 100644 index 0000000..b2ba62d --- /dev/null +++ b/doc/tutorial-data/untypable-json/untypable_v1.atd @@ -0,0 +1,14 @@ +(* File untypable.atd *) + +type json <ocaml module="Yojson.Safe"> = abstract + (* uses type Yojson.Safe.t, + with the functions Yojson.Safe.write_json + and Yojson.Safe.read_json *) + +type obj_list = obj list + +type obj = { + ?label: string option; + ?labels: string list option; + value: json +} diff --git a/doc/tutorial-data/untypable-json/untypable_v2.atd b/doc/tutorial-data/untypable-json/untypable_v2.atd new file mode 100644 index 0000000..eceed6e --- /dev/null +++ b/doc/tutorial-data/untypable-json/untypable_v2.atd @@ -0,0 +1,12 @@ +type raw_json <ocaml module="Yojson.Safe" t="json"> = abstract + (* uses type Yojson.Safe.t, + with the functions Yojson.Safe.write_json + and Yojson.Safe.read_json *) + +type obj_list = obj list + +type obj = { + ?label: string option; + ?labels: string list option; + value: raw_json +} diff --git a/doc/tutorial-data/validate/demo.sh b/doc/tutorial-data/validate/demo.sh new file mode 100755 index 0000000..dc92c77 --- /dev/null +++ b/doc/tutorial-data/validate/demo.sh @@ -0,0 +1,20 @@ +#! /bin/sh -e + +set -x +cat resume.atd +atdgen -t resume.atd +atdgen -j resume.atd +atdgen -v resume.atd +ls +ocamlfind ocamlc -c resume_t.mli -package atdgen +ocamlfind ocamlc -c resume_v.mli -package atdgen +ocamlfind ocamlc -c resume_j.mli -package atdgen +ocamlfind ocamlopt -c resume_t.ml -package atdgen +ocamlfind ocamlopt -c resume_util.ml -package atdgen +ocamlfind ocamlopt -c resume_v.ml -package atdgen +ocamlfind ocamlopt -c resume_j.ml -package atdgen +ocamlfind ocamlopt -c resume.ml -package atdgen +ocamlfind ocamlopt -o test_resume \ + resume_t.cmx resume_util.cmx resume_v.cmx resume_j.cmx resume.cmx \ + -package atdgen -linkpkg +./test_resume diff --git a/doc/tutorial-data/validate/resume.atd b/doc/tutorial-data/validate/resume.atd new file mode 100644 index 0000000..3adf81f --- /dev/null +++ b/doc/tutorial-data/validate/resume.atd @@ -0,0 +1,16 @@ +type text = string <ocaml valid="Resume_util.validate_some_text"> + +type date = { + year : int; + month : int; + day : int; +} <ocaml valid="Resume_util.validate_date"> + +type job = { + company : text; + title : text; + start_date : date; + ?end_date : date option; +} <ocaml valid="Resume_util.validate_job"> + +type work_experience = job list diff --git a/doc/tutorial-data/validate/resume.ml b/doc/tutorial-data/validate/resume.ml new file mode 100644 index 0000000..ea0b32f --- /dev/null +++ b/doc/tutorial-data/validate/resume.ml @@ -0,0 +1,31 @@ +let check_experience x = + let is_valid = match Resume_v.validate_work_experience [] x with + | None -> false + | _ -> true + in + Printf.printf "%s:\n%s\n" + (if is_valid then "VALID" else "INVALID") + (Yojson.Safe.prettify (Resume_j.string_of_work_experience x)) + +let () = + (* one valid date *) + let valid = { Resume_t.year = 2000; month = 2; day = 29 } in + (* one invalid date *) + let invalid = { Resume_t.year = 2010; month = 0; day = 0 } in + (* two more valid dates, created with Resume_v.create_date *) + let date1 = { Resume_t.year = 2005; month = 8; day = 1 } in + let date2 = { Resume_t.year = 2006; month = 3; day = 22 } in + + let job = { + Resume_t.company = "Acme Corp."; + title = "Tester"; + start_date = date1; + end_date = Some date2; + } + in + let valid_job = { job with Resume_t.start_date = valid } in + let invalid_job = { job with Resume_t.end_date = Some invalid } in + let valid_experience = [ job; valid_job ] in + let invalid_experience = [ job; invalid_job ] in + check_experience valid_experience; + check_experience invalid_experience diff --git a/doc/tutorial-data/validate/resume_util.ml b/doc/tutorial-data/validate/resume_util.ml new file mode 100644 index 0000000..631c6a3 --- /dev/null +++ b/doc/tutorial-data/validate/resume_util.ml @@ -0,0 +1,52 @@ +open Resume_t + +let ascii_printable c = + let n = Char.code c in + n >= 32 && n <= 127 + +(* + Check that string is not empty and contains only ASCII printable + characters (for the sake of the example; we use UTF-8 these days) +*) +let validate_some_text s = + s <> "" && + try + String.iter (fun c -> if not (ascii_printable c) then raise Exit) s; + true + with Exit -> + false + +(* + Check that the combination of year, month and day exists in the + Gregorian calendar. +*) +let validate_date x = + let y = x.year in + let m = x.month in + let d = x.day in + m >= 1 && m <= 12 && d >= 1 && + (let dmax = + match m with + 2 -> + if y mod 4 = 0 && not (y mod 100 = 0) || y mod 400 = 0 then 29 + else 28 + | 1 | 3 | 5 | 7 | 8 | 10 | 12 -> 31 + | _ -> 30 + in + d <= dmax) + +(* Compare dates chronologically *) +let compare_date a b = + let c = compare a.year b.year in + if c <> 0 then c + else + let c = compare a.month b.month in + if c <> 0 then c + else compare a.day b.day + +(* Check that the end_date, when defined, is not earlier than the start_date *) +let validate_job x = + match x.end_date with + None -> true + | Some end_date -> + compare_date x.start_date end_date <= 0 diff --git a/doc/tutorial.rst b/doc/tutorial.rst index db44426..50660ee 100644 --- a/doc/tutorial.rst +++ b/doc/tutorial.rst @@ -1,6 +1,6 @@ -********************* -Tutorial Introduction -********************* +******** +Tutorial +******** What is atdgen? =============== @@ -9,7 +9,7 @@ Atdgen is a tool that derives OCaml boilerplate code from type definitions. Currently it provides support for: * `JSON <http://json.org/>`_ serialization and deserialization. -* `Biniou <https://mjambon.github.io/atdgen-doc/biniou-format.txt>`_ serialization and deserialization. +* `Biniou <https://raw.githubusercontent.com/mjambon/atd/master/atdgen-doc/src/biniou-format.txt>`_ serialization and deserialization. Biniou is a binary format extensible like JSON but more compact and faster to process. * Convenience functions for creating and validating OCaml data. @@ -29,13 +29,12 @@ which was based on Camlp4: * runs fast, keeping build times low. * same ATD definitions can be used to generate code other than OCaml. See for instance - `atdj <https://github.com/esperco/atdj>`_ + `atdj <https://github.com/mjambon/atd/tree/master/atdj>`_ which generates Java classes for JSON IO. Auto-generating GUI widgets from type definitions is another popular use of annotated type definitions. The implementation of such code generators is facilitated by the - `\`atd\` <https://github.com/mjambon/atd>`_ library. - + `atd <https://github.com/mjambon/atd/tree/atd>`_ library. Prerequisites @@ -119,7 +118,7 @@ And finally we run our `hello` program: $ ./hello {"year":1970,"month":1,"day":1} -`Source code for this section <https://github.com/mjambon/atdgen-doc/src/tutorial-data/hello>`_ +`Source code for this section <https://github.com/mjambon/doc/src/tutorial-data/hello>`_ Inspecting and pretty-printing JSON =================================== @@ -204,7 +203,7 @@ We now compile and run prettify.ml: } ] -`Source code for this section <https://github.com/mjambon/atdgen-doc/src/tutorial-data/pretty-json>`__ +`Source code for this section <https://github.com/mjambon/doc/src/tutorial-data/pretty-json>`__ @@ -349,7 +348,7 @@ anymore (for this user on this machine). The following now works: 4, <"Node": (<"Empty">, 5, <"Empty">)>)>)> -`Source code for this section <https://github.com/mjambon/atdgen-doc/src/tutorial-data/inspect-biniou>`__ +`Source code for this section <https://github.com/mjambon/doc/src/tutorial-data/inspect-biniou>`__ Optional fields and default values ================================== @@ -547,38 +546,79 @@ example is: type t = string <ocaml valid="fun s -> String.length s >= 8"> option -``atdgen -v`` will produce something equivalent to the following -implementation: +As we can see from this example, the validation function is specified using the +annotation ``<ocaml valid="p">``, where ``p`` is a predicate ``p : t -> bool``, +returning ``true`` when the value of type ``t`` is valid and ``false`` +otherwise. + +Calling ``atdgen -v`` on a file containing this specification will produce +a validation function equivalent to the following implementation: .. code-block:: ocaml - let validate_t x = + let validate_t path x = match x with - None -> true - | Some x -> (fun s -> String.length s >= 8) x + | None -> None + | Some x -> + let msg = "Failed check by fun s -> String.length s >= 8" in + if (fun s -> String.length s >= 8) x + then None + else Some {error_path = path; error_msg = msg} -Let's now consider a more realistic example with complex validators defined in a -separate ``.ml`` file. We created the following 3 source files: +Let's consider this particular example as an illustration of the general shape +of generated validation functions. -* ``resume.atd``: contains the type definitions with annotations -* ``resume_util.ml``: contains our handwritten validators -* ``resume.ml``: is our main program that creates data and - calls the validators +The function takes two arguments: the first, ``path``, is a list indicating +where the second, ``x``, was encountered. As specified by our example ``.atd`` +code above, ``x`` has type ``t option``. +The body of the validation function does two things: -In terms of OCaml modules we have: +1. it checks the value of ``x`` against the validation function specified in our +``.atd`` file, namely, checking whether there is ``Some s``, and verifying that +``s`` is at least 8 characters long if so +2. in the event that the validation check fails, it constructs an appropriate +error record. +In general, generated validation functions for a type ``t`` have a type +equivalent to ``validate_t : path -> t -> error option``, where the ``path`` +gives the current location in a data structure and the ``error`` is a record of +the location of, and reason for, validation failure. -* ``Resume_t``: produced by ``atdgen -t resume.atd``, provides OCaml type - definitions -* ``Resume_util``: depends on ``Resume_t``, provides validators mentioned in - ``resume.atd`` -* ``Resume_v``: produced by ``atdgen -v resume.atd``, depends on - ``Resume_util``, provides a validator for each type -* ``Resume``: depends on ``Resume_v``, uses the validators +A return value of ``None`` indicates successful validation, while ``Some +{error_path; error_msg}`` tells us where and why validation failed. +Let's now consider a more realistic example with complex validators defined in a +separate ``.ml`` file. We will define a data structure representing a section of +a resume recording work experience. We will also define validation functions +that can enforce certain properties to protect against errors and junk data. + +In the course of this example, we will manually create the following 3 source +files: -Type definitions are placed in ``resume.atd``: +* ``resume.atd``: contains the type definitions with annotations +* ``resume_util.ml``: contains our handwritten validators +* ``resume.ml``: is our main program that creates data and checks it using our + generated validation functions. + +After generating additional code with ``atdgen``, we will end up with the +following OCaml modules: + +* ``Resume_t``: generated into ``resume_t.ml`` by ``atdgen -t resume.atd``, this + provides our OCaml type definitions +* ``Resume_util``: written manually in ``resume_util.ml``, this depends on + ``Resume_t`` and provides validators we will use in ``resume.atd`` +* ``Resume_v``: generated into ``resume_v.ml`` by ``atdgen -v resume.atd``, this + depends on ``Resume_util`` and ``Resume_t`` and provides a validation function + for each type +* ``Resume_j``: generated into ``resume_j.ml`` by ``atdgen -j resume.atd``, this + provides functions to serialize and deserialize data in and out of JSON. +* ``Resume``: written manually in ``resume.ml``, this depends on ``Resume_v``, + and ``Resume_t``, and makes use of the generated types and validation + functions. + +To begin, we specify type definitions for a data structure representing a resume +in ``resume.atd``: .. code-block:: ocaml @@ -599,7 +639,10 @@ Type definitions are placed in ``resume.atd``: type work_experience = job list -``resume_util.ml`` contains our handwritten validators: +We can now call ``atdgen -t resume.atd`` to generate our ``Resume_t`` module in +``resume_t.ml``, providing our data types. Using these data types, we'll define +the following handwritten validators in ``resume_util.ml`` (note that we've +already referred to these validators in ``resume.atd``): .. code-block:: ocaml @@ -656,13 +699,18 @@ Type definitions are placed in ``resume.atd``: | Some end_date -> compare_date x.start_date end_date <= 0 -``resume.ml`` uses the ``validate_work_experience`` function provided by the -``Resume_v`` module: +After we call ``atdgen -v resume.atd``, the module ``Resume_v`` will be +generated in ``resume_v.ml``, providing the function +``validate_work_experience`` . We can then use this function, along with the +generated ``Resume_j`` in the following program written in ``resume.ml``: .. code-block:: ocaml let check_experience x = - let is_valid = Resume_v.validate_work_experience x in + let is_valid = match Resume_v.validate_work_experience [] x with + | None -> false + | _ -> true + in Printf.printf "%s:\n%s\n" (if is_valid then "VALID" else "INVALID") (Yojson.Safe.prettify (Resume_j.string_of_work_experience x)) @@ -724,7 +772,7 @@ Output: "end_date": { "year": 1900, "month": 0, "day": 0 } } -`Source code for this section <https://github.com/mjambon/atdgen-doc/src/tutorial-data/validate>`__ +`Source code for this section <https://github.com/mjambon/atd/tree/master/doc/tutorial-data/validate>`__ Modularity: referring to type definitions from another ATD file =============================================================== @@ -784,7 +832,7 @@ Output: {"name":"foo","data":[{"x":1,"y":2},{"x":3,"y":4}]} -`Source code for this section <https://github.com/mjambon/atdgen-doc/src/tutorial-data/modularity>`__ +`Source code for this section <https://github.com/mjambon/doc/src/tutorial-data/modularity>`__ Managing JSON configuration files @@ -1033,7 +1081,7 @@ program called `config`: let () = main () The full source code for this section with examples can be inspected -and `downloaded here <https://github.com/mjambon/atdgen-doc/src/tutorial-data/config-file>`__. +and `downloaded here <https://github.com/mjambon/doc/src/tutorial-data/config-file>`__. Integration with ocamldoc @@ -1132,7 +1180,7 @@ and ``.ml`` produced by ``atdgen``. GNU Make -------- -We provide `\`Atdgen.mk\` <https://github.com/mjambon/atdgen-make>`__, a generic +We provide `Atdgen.mk <https://github.com/mjambon/atdgen-make>`__, a generic makefile that defines the dependencies and rules for generating OCaml ``.mli`` and ``.ml`` files from ``.atd`` files containing type definitions. The ``Atdgen.mk`` file contains its own documentation. @@ -1178,8 +1226,8 @@ Ocamlbuild There is an `atdgen plugin for ocamlbuild <https://github.com/hcarty/ocamlbuild-plugins/blob/master/myatdgen.ml>`__. -Dune (jbuilder) ---------------- +Dune (formerly jbuilder) +------------------------ Dune currently needs atdgen build rules specified manually. Given an ``example.atd``, this will usually look like: @@ -1187,16 +1235,16 @@ this will usually look like: .. code-block:: scheme (rule - ((targets (example_j.ml - example_j.mli)) - (deps (example.atd)) - (action (run atdgen -j -j-std ${^})))) + (targets example_j.ml + example_j.mli) + (deps example.atd) + (action (run atdgen -j -j-std %{deps}))) (rule - ((targets (example_t.ml - example_t.mli)) - (deps (example.atd)) - (action (run atdgen -t ${^})))) + (targets example_t.ml + example_t.mli) + (deps example.atd) + (action (run atdgen -t %{deps}))) You can refer to ``example_t.ml`` and ``example_j.ml`` as usual (by default, they will be automatically linked into the library being built in the same directory). @@ -1249,7 +1297,7 @@ or a field ``labels`` of type ``string``: (* File untypable.atd *) type json <ocaml module="Yojson.Safe"> = abstract - (* uses type Yojson.Safe.json, + (* uses type Yojson.Safe.t, with the functions Yojson.Safe.write_json and Yojson.Safe.read_json *) @@ -1268,7 +1316,7 @@ in the annotation, i.e.: .. code-block:: ocaml type raw_json <ocaml module="Yojson.Safe" t="json"> = abstract - (* uses type Yojson.Safe.json, + (* uses type Yojson.Safe.t, with the functions Yojson.Safe.write_json and Yojson.Safe.read_json *) |