1# API File Generation
2
3There are certain pieces of `NeuralNetworksTypes.h`, `Types.h`,
4`OperandTypes.h`, `OperationTypes.h`, and of our various `*.hal`/`*.aidl` files
5that ought to be kept in sync -- most notably the operand type and operation
6type definitions and descriptions. To avoid having to do this manually, a tool
7`generate_api.py` is employed to combine a single *specification file* with one
8*template file* per API file to produce that API file. The
9script `generate_api.sh` invokes `generate_api.py` once per API file, passing
10appropriate arguments.
11
12## `generate_api.sh`
13
14The environment variable `ANDROID_BUILD_TOP` must be set.
15
16Invoked with no arguments or with the `--mode=update` argument, this script
17regenerates each API file in place, by invoking `generate_api.py` once per
18generated file.
19
20Invoked with the `--mode=hook` argument, this script checks whether NDK and
21Canonical files need to be regenerated.
22
23When the `--dryrun` argument is present, this script shows how it would invoke
24`generate_api.py` but does not actually regenerate files or check whether they
25need to be regenerated.
26
27## `generate_api.py`
28
29This tool generates a single output file from an input specification file and an
30input template file.  It takes the following mandatory arguments:
31
32* `--output OUTPUT` path to generated output file (such as `Types.h`)
33* `--specification SPECIFICATION` path to input specification file
34* `--template TEMPLATE` path to input template file
35* `--kind KIND` token identifying kind of file to generate
36
37The "kind" is an arbitrary token that the specification file can reference with
38the `%kind` directive to help generate different text in different situations.
39It has no meaning to the tool itself.  Today, the following kinds are used:
40`ndk` (when generating `NeuralNetworksTypes.h`), `canonical` (when generating
41`Types.h`, `OperandTypes.h`, and `OperationTypes.h`), `hal_1.0` (when generating
42`1.0/types.hal`), `hal_1.1`, `hal_1.2`, `hal_1.3` and `aidl` (when
43generating `OperandType.aidl` and `OperationType.aidl`).
44
45## Template File Syntax
46
47Every line of the template file is copied verbatim to the output file *unless*
48that line begins with `%`.
49
50A line that begins with `%%` is a comment, and is ignored.
51
52A line that begins with `%` and is not a comment is a *directive*.
53
54### Directives
55
56#### `%insert *name*`
57
58Copy the *section* with the specified *name* from the specification file to the
59output file.  The section is defined by a `%section` directive in the
60specification file.
61
62#### `%insert-indented *count* *name*`
63
64Similar to `%insert *name*`, but each non-empty copied line is prefixed with
65*count* space characters.  *count* must be a non-negative integer.
66
67## Specification File Syntax
68
69The specification file consists of comments, *directives*, and other text.
70
71A line that begins with `%%` is a comment, and is ignored.
72
73A line that begins with `%` and is not a comment is a *directive*.
74
75The meaning of a line that is neither a comment nor a directive depends on the
76context -- the *region* in which that line appears.
77
78### Regions
79
80The specification file is divided into *regions*, which are sequences of lines
81delimited by certain directives.
82
83Certain regions can enclose certain other regions, but this is very limited:
84
85* A conditional region can enclose a section region.
86* A section region can enclose a conditional region.
87
88Equivalently:
89
90* A conditional region can be enclosed by a section region.
91* A section region can be enclosed by a conditional region.
92
93#### null region
94
95A *null region* is a sequence of lines that is not part of any other region.
96For example, a specification file that contains no directives other than
97`%define` and `%define-kinds` consists of a single null region.
98
99Within a null region, all lines other than directives are treated as comments
100and are ignored.
101
102#### conditional region
103
104A *conditional region* is a sequence of lines immediately preceded by the `%kind
105*list*` directive and immediately followed by the `%/kind` directive.  The
106`%kind` directive establishes a condition state **on** or **off** (see the
107description of the directive for details).  When the condition is **on**, the
108lines in the region are processed normally (i.e., directives have their usual
109effect, and non-directive lines are added to the enclosing section region, if
110any).  When the condition is **off**, lines in the region other than the `%else`
111directive are ignored *except* that even ignored directives undergo some level
112of syntactic and semantic checking.
113
114#### section region
115
116A *section region* is a sequence of lines immediately preceded by the `%section
117*name*` directive and immediately followed by the `%/section` directive.  Every
118line in the sequence that doesn't begin with `%` undergoes macro substitution,
119and the resulting lines are associated with the section name.  They can be
120inserted into the generated output file as directed by the template file's
121`%insert` and `%insert-indented` directives.  They can be added to another
122section region with the with the specification file's `%insert` and
123`%insert-indented` directives.
124
125This is the mechanism by which a specification file contributes text to the
126generated output file.
127
128### Directives
129
130#### `%define *name* *body*`
131
132Defines a macro identified by the token *name*.  The *body* is separated from
133the *name* by exactly one whitespace character, and extends to the end of the
134line -- it may contain whitespace itself.  For example,
135
136  %define test  this body begins and ends with a space character
137
138Macro substitution occurs within a section region: a substring `%{*name*}` is
139replaced with the corresponding *body*.  Macro substitution is *not* recursive:
140A substring `%{*name2*}` in *body* will not undergo macro substitution, except
141as discussed for *macro arguments* below.
142
143Permitted in regions: null, conditional, section
144
145##### macro arguments
146
147The more general form of a macro invocation is `%{*name* *arglist*}`, where
148*arglist* is a list of whitespace-separated arguments.  Within the *body*, a
149substring of the form `%{argnum}` will be replaced by the corresponding argument
150from *arglist*.  For example, if the definition is
151
152```
153%define test second is %{2}, first is %{1}
154```
155
156then the macro invocation
157
158```
159%{test alpha beta}
160```
161
162is expanded to
163
164```
165second is beta, first is alpha
166```
167
168The only check on the number of arguments supplied at macro invocation time is
169that there must be at least as many arguments as the highest `%{argnum}`
170reference in the macro body.  In the example above, `%{test alpha}` would be an
171error, but `%{test alpha beta gamma}` would not.
172
173#### `%insert *name*`
174
175Adds all lines from the named section region to the current section region.
176
177Permitted in regions: section
178
179#### `%insert-indented *count* *name*`
180
181Similar to `%insert *name*`, but each non-empty added line is prefixed
182with *count* space characters.  *count* must be a non-negative integer.
183
184Permitted in regions: section
185
186#### `%kind *list*`, `%else`, `%/kind`
187
188`%kind *list*` creates a *conditional region* terminated by `%/kind`.
189
190The *list* consists of a space-delimited list of tokens, any of which may end in
191`*` to indicate a *wildcard pattern* or `+` to indicate a *lowest version
192pattern*.  Any other pattern is a *simple pattern*.  The condition is **on** in
193three cases:
194* One of the simple pattern tokens equals the "kind"
195* One of the wildcard pattern tokens less the `*` is a prefix of the "kind"
196* One of the lowest version pattern tokens less the `+` matches the "kind" or
197  the "kind" matches any token to the right from the lowest version pattern in
198  the list passed to %define-kinds
199
200In all other cases, the condition is **off**.
201
202Within the region, the condition is inverted every time the `%else` directive
203appears.
204
205Permitted in regions: null, section
206
207#### `%define-kinds *list*`
208
209This directive has two purposes:
210
211* Validity-checking.  If the "kind" is not on the space-delimited *list* of tokens,
212  `generate_api.py` terminates with an error.
213* Ordering the possible kinds for the *lowest version pattern* (see the section
214  above for the explanation of the pattern).
215
216Only one such directive is allowed per specification file.
217
218Permitted in regions: null, section
219
220#### `%section *name*`, `%/section`
221
222`%section *name*` creates a *section region* terminated by `%/section`.
223
224Permitted in regions: null, conditional
225