1# Metalava 2 3Metalava is a metadata generator intended for JVM type projects. The main 4users of this tool are Android Platform and AndroidX libraries, however this 5tool also works on non-Android libraries. 6 7Metalava has many features related to API management. Some examples of the most 8commonly used ones are: 9 10* Allows extracting the API (into signature text files, into stub API files 11 which in turn get compiled into android.jar, the Android SDK library) and 12 more importantly to hide code intended to be implementation only, driven by 13 javadoc comments like @hide, @doconly, @removed, etc, as well as various 14 annotations. 15 16* Extracting source level annotations into external annotations file (such as 17 the typedef annotations, which cannot be stored in the SDK as .class level 18 annotations) to ship alongside the Android SDK and used by Android Lint. 19 20* Diffing versions of the API and determining whether a newer version is 21 compatible with the older version. (See [COMPATIBILITY.md](COMPATIBILITY.md)) 22 23## Building and running 24 25To download the code and any dependencies required for building, see [DOWNLOADING.md](DOWNLOADING.md) 26 27To build: 28 29 $ cd tools/metalava 30 $ ./gradlew 31 32It puts build artifacts in `../../out/metalava/`. 33 34To run the metalava executable: 35 36### Through Gradle 37 38To list all the options: 39 40 $ ./gradlew run 41 42To run it with specific arguments: 43 44 $ ./gradlew run --args="--api path/to/api.txt" 45 46### Through distribution artifact 47 48First build it with: 49 50 $ ./gradlew installDist 51 52Then run it with: 53 54 $ ../../out/metalava/metalava/build/install/metalava/bin/metalava 55 _ _ 56 _ __ ___ ___| |_ __ _| | __ ___ ____ _ 57 | '_ ` _ \ / _ \ __/ _` | |/ _` \ \ / / _` | 58 | | | | | | __/ || (_| | | (_| |\ V / (_| | 59 |_| |_| |_|\___|\__\__,_|_|\__,_| \_/ \__,_| 60 61 metalava extracts metadata from source code to generate artifacts such as the 62 signature files, the SDK stub files, external annotations etc. 63 64 Usage: metalava <flags> 65 66 Flags: 67 68 --help This message. 69 --quiet Only include vital output 70 --verbose Include extra diagnostic output 71 72 ... 73 74(*output truncated*) 75 76### Maven artifacts 77 78To build Metalava's Maven artifacts including `.pom` and `.module` metadata, run: 79 80 $ ./gradlew createArchive 81 82Then locate the artifacts under `../../out/dist/repo/m2repository`. 83 84### Integration testing 85 86To build and run Metalava against a pinned version of an AndroidX library you can 87run the following: 88 89 $ INTEGRATION=true ./gradlew integration:run 90 91Details on what runs are in `integration/build.gradle.kts`. 92 93It can also be run for repeated measurement using [gradle-profiler](https://github.com/gradle/gradle-profiler) with 94 95 $ INTEGRATION=true /path/to/gradle-profiler --benchmark --project-dir . --scenario-file integration/integration.scenarios 96 97## Features 98 99* Ability to read in an existing android.jar file instead of from source, which 100 means we can regenerate signature files etc for older versions according to 101 new formats (e.g. to fix past errors in doclava, such as annotation instance 102 methods which were accidentally not included.) 103 104* Ability to merge in data (annotations etc) from external sources, such as 105 IntelliJ external annotations data as well as signature files containing 106 annotations. This isn't just merged at export time, it's merged at codebase 107 load time such that it can be part of the API analysis. 108 109* Support for an updated signature file format (which is described in [FORMAT.md](FORMAT.md)) 110 111 * Address errors in the doclava1 format which for example was missing 112 annotation class instance methods 113 114 * Improve the signature format such that it for example labels enums "enum" 115 instead of "abstract class extends java.lang.Enum", annotations as 116 "@interface" instead of "abstract class extends java.lang.Annotation", sorts 117 modifiers in the canonical modifier order, using "extends" instead of 118 "implements" for the superclass of an interface, and many other similar 119 tweaks outlined in the `Compatibility` class. (Metalava also allows (and 120 ignores) block comments in the signature files.) 121 122 * Add support for writing (and reading) annotations into the signature 123 files. This is vital now that some of these annotations become part of the 124 API contract (in particular nullness contracts, as well as parameter names 125 and default values.) 126 127 * Support for a "compact" nullness format -- one based on Kotlin's 128 syntax. Since the goal is to have **all** API elements explicitly state 129 their nullness contract, the signature files would very quickly become 130 bloated with @NonNull and @Nullable annotations everywhere. So instead, the 131 signature format now uses a suffix of `?` for nullable, `!` for not yet 132 annotated, and nothing for non-null. 133 134 Instead of 135 136 method public java.lang.Double convert0(java.lang.Float); 137 method @Nullable public java.lang.Double convert1(@NonNull java.lang.Float); 138 139 we have 140 141 method public java.lang.Double! convert0(java.lang.Float!); 142 method public java.lang.Double? convert1(java.lang.Float); 143 144 * Other compactness improvements: Skip packages in some cases both for export 145 and reinsert during import. Specifically, drop "java.lang." from package 146 names such that you have 147 148 method public void onUpdate(int, String); 149 150 instead of 151 152 method public void onUpdate(int, java.lang.String); 153 154 Similarly, annotations (the ones considered part of the API; unknown 155 annotations are not included in signature files) use just the simple name 156 instead of the full package name, e.g. `@UiThread` instead of 157 `@android.annotation.UiThread`. 158 159 * Misc documentation handling; for example, it attempts to fix sentences that 160 javadoc will mistreat, such as sentences that "end" with "e.g. ". It also 161 looks for various common typos and fixes those; here's a sample error 162 message running metalava on master: Enhancing docs: 163 164 frameworks/base/core/java/android/content/res/AssetManager.java:166: error: Replaced Kitkat with KitKat in documentation for Method android.content.res.AssetManager.getLocales() [Typo] 165 frameworks/base/core/java/android/print/PrinterCapabilitiesInfo.java:122: error: Replaced Kitkat with KitKat in documentation for Method android.print.PrinterCapabilitiesInfo.Builder.setColorModes(int, int) [Typo] 166 167* Built-in support for injecting new annotations for use by the Kotlin compiler, 168 not just nullness annotations found in the source code and annotations merged 169 in from external sources, but also inferring whether nullness annotations have 170 recently changed and if so marking them as @Migrate (which lets the Kotlin 171 compiler treat errors in the user code as warnings instead of errors.) 172 173* Support for generating documentation into the stubs files (so we can run 174 javadoc or [Dokka](https://github.com/Kotlin/dokka) on the stubs files instead 175 of the source code). This means that the documentation tool itself does not 176 need to be able to figure out which parts of the source code is included in 177 the API and which one is implementation; it is simply handed the filtered API 178 stub sources that include documentation. 179 180* Support for parsing Kotlin files. API files can now be implemented in Kotlin 181 as well and metalava will parse and extract API information from them just as 182 is done for Java files. 183 184* Like doclava1, metalava can diff two APIs and warn about API compatibility 185 problems such as removing API elements. Metalava adds new warnings around 186 nullness, such as attempting to change a nullness contract incompatibly 187 (e.g. you can change a parameter from non null to nullable for final classes, 188 but not versa). It also lets you diff directly on a source tree; it does not 189 require you to create two signature files to diff. 190 191* Consistent stubs: In doclava1, the code which iterated over the API and 192 generated the signature files and generated the stubs had diverged, so there 193 was some inconsistency. In metalava the stub files contain **exactly** the 194 same signatures as in the signature files. 195 196 (This turned out to be incredibly important; this revealed for example that 197 StringBuilder.setLength(int) was missing from the API signatures since it is a 198 public method inherited from a package protected super class, which the API 199 extraction code in doclava1 missed, but accidentally included in the SDK 200 anyway since it packages package private classes. Metalava strictly applies 201 the exact same API as is listed in the signature files, and once this was 202 hooked up to the build it immediately became apparent that it was missing 203 important methods that should really be part of the API.) 204 205* API Lint: Metalava can optionally (with --api-lint) run a series of additional 206 checks on the public API in the codebase and flag issues that are discouraged 207 or forbidden by the Android API Council; there are currently around 80 checks. 208 Some of these take advantage of looking at the source code which wasn't 209 possible with the signature-file based Python version; for example, it looks 210 inside method bodies to see if you're synchronizing on this or the current 211 class, which is forbidden. 212 213* Baselines: Metalava can report all of its issues into a "baseline" file, which 214 records the current set of issues. From that point forward, when metalava 215 finds a problem, it will only be reported if it is not already in the 216 baseline. This lets you enforce new issues going forward without having to 217 fix all existing violations. Periodically, as older issues are fixed, you can 218 regenerate the baseline. For issues with some false positives, such as API 219 Lint, being able to check in the set of accepted or verified false positives 220 is quite important. 221 222* Metalava can generate reports about nullness annotation coverage (which helps 223 target efforts since we plan to annotate the entire API). First, it can 224 generate a raw count: 225 226 Nullness Annotation Coverage Statistics: 227 1279 out of 46900 methods were annotated (2%) 228 2 out of 21683 fields were annotated (0%) 229 2770 out of 47492 parameters were annotated (5%) 230 231 More importantly, you can also point it to some existing compiled applications 232 (.class or .jar files) and it will then measure the annotation coverage of the 233 APIs used by those applications. This lets us target the most important APIs 234 that are currently used by a corpus of apps and target our annotation efforts 235 in a targeted way. For example, running the analysis on the current version of 236 framework, and pointing it to the 237 [Plaid](https://github.com/nickbutcher/plaid) app's compiled output with 238 239 ... --annotation-coverage-of ~/plaid/app/build/intermediates/classes/debug 240 241 This produces the following output: 242 243 324 methods and fields were missing nullness annotations out of 650 total 244 API references. API nullness coverage is 50% 245 246 ``` 247 | Qualified Class Name | Usage Count | 248 |--------------------------------------------------------------|-----------------:| 249 | android.os.Parcel | 146 | 250 | android.view.View | 119 | 251 | android.view.ViewPropertyAnimator | 114 | 252 | android.content.Intent | 104 | 253 | android.graphics.Rect | 79 | 254 | android.content.Context | 61 | 255 | android.widget.TextView | 53 | 256 | android.transition.TransitionValues | 49 | 257 | android.animation.Animator | 34 | 258 | android.app.ActivityOptions | 34 | 259 | android.view.LayoutInflater | 31 | 260 | android.app.Activity | 28 | 261 | android.content.SharedPreferences | 26 | 262 | android.content.SharedPreferences.Editor | 26 | 263 | android.text.SpannableStringBuilder | 23 | 264 | android.view.ViewGroup.MarginLayoutParams | 21 | 265 | ... (99 more items | | 266 ``` 267 268Top referenced un-annotated members: 269 270 ``` 271 | Member | Usage Count | 272 |--------------------------------------------------------------|-----------------:| 273 | Parcel.readString() | 62 | 274 | Parcel.writeString(String) | 62 | 275 | TextView.setText(CharSequence) | 34 | 276 | TransitionValues.values | 28 | 277 | View.getContext() | 28 | 278 | ViewPropertyAnimator.setDuration(long) | 26 | 279 | ViewPropertyAnimator.setInterpolator(android.animation.Ti... | 26 | 280 | LayoutInflater.inflate(int, android.view.ViewGroup, boole... | 23 | 281 | Rect.left | 22 | 282 | Rect.top | 22 | 283 | Intent.Intent(android.content.Context, Class<?>) | 21 | 284 | Rect.bottom | 21 | 285 | TransitionValues.view | 21 | 286 | VERSION.SDK_INT | 18 | 287 | Context.getResources() | 18 | 288 | EditText.getText() | 18 | 289 | ... (309 more items | | 290 ``` 291 292 From this it's clear that it would be useful to start annotating 293 android.os.Parcel and android.view.View for example where there are 294 unannotated APIs that are frequently used, at least by this app. 295 296* Built on top of a full, type-resolved AST. Doclava1 was integrated with 297 javadoc, which meant that most of the source tree was opaque. Therefore, as 298 just one example, the code which generated documentation for typedef constants 299 had to require the constants to all share a single prefix it could look 300 for. However, in metalava, annotation references are available at the AST 301 level, so it can resolve references and map them back to the original field 302 references and include those directly. 303 304* Support for extracting annotations. Metalava can also generate the external 305 annotation files needed by Studio and lint in Gradle, which captures the 306 typedefs (@IntDef and @StringDef classes) in the source code. Prior to this 307 this was generated manually via the development/tools/extract code. This also 308 merges in manually curated data; some of this is in the manual/ folder in this 309 project. 310 311* Support for extracting API levels (api-versions.xml). This was generated by 312 separate code (tools/base/misc/api-generator), invoked during the build. This 313 functionality is now rolled into metalava, which has one very important 314 attribute: metalava will use this information when recording API levels for 315 API usage. (Prior to this, this was based on signature file parsing in 316 doclava, which sometimes generated incorrect results. Metalava uses the 317 android.jar files themselves to ensure that it computes the exact available 318 SDK data for each API level.) 319 320* Misc other features. For example, if you use the @VisibleForTesting annotation 321 from the support library, where you can express the intended visibility if the 322 method had not required visibility for testing, then metalava will treat that 323 method using the intended visibility instead when generating signature files 324 and stubs. 325 326## Architecture & Implementation 327 328Metalava is implemented on top of IntelliJ parsing APIs (PSI and UAST). However, 329these are hidden behind a "model": an abstraction layer which only exposes high 330level concepts like packages, classes and inner classes, methods, fields, and 331modifier lists (including annotations). 332 333This is done for multiple reasons: 334 335(1) It allows us to have multiple "back-ends": for example, metalava can read in 336 a model not just from parsing source code, but from reading older SDK 337 android.jar files (e.g. backed by bytecode) or reading previous signature 338 files. Reading in multiple versions of an API lets doclava perform 339 "diffing", such as warning if an API is changing in an incompatible way. It 340 can also generate signature files in the new format (including data that was 341 missing in older signature files, such as annotation methods) without having 342 to parse older source code which may no longer be easy to parse. 343 344(2) There's a lot of logic for deciding whether code found in the source tree 345 should be included in the API. With the model approach we can build up an 346 API and for example mark a subset of its methods as included. By having a 347 separate hierarchy we can easily perform this work once and pass around our 348 filtered model instead of passing around PsiClass and PsiMethod instances 349 and having to keep the filtered data separately and remembering to always 350 consult the filter, not the PSI elements directly. 351 352The basic API element class is "Item". (In doclava1 this was called a 353"DocInfo".) There are several sub interfaces of Item: PackageItem, ClassItem, 354MemberItem, MethodItem, FieldItem, ParameterItem, etc. And then there are 355several implementation hierarchies: One is PSI based, where you point metalava 356to a source tree or a .jar file, and it constructs Items built on top of PSI: 357PsiPackageItem, PsiClassItem, PsiMethodItem, etc. Another is textual, based on 358signature files: TextPackageItem, TextClassItem, and so on. 359 360The "Codebase" class captures a complete API snapshot (including classes that 361are hidden, which is why it's called a "Codebase" rather than an "API"). 362 363There are methods to load codebases - from source folders, from a .jar file, 364from a signature file. That's how API diffing is performed: you load two 365codebases (from whatever source you want, typically a previous API signature 366file and the current set of source folders), and then you "diff" the two. 367 368There are several key helpers that help with the implementation, detailed next. 369 370### Visiting Items 371 372First, metalava provides an ItemVisitor. This lets you visit the API easily. 373For example, here's how you can visit every class: 374 375 codebase.accept(object : ItemVisitor() { 376 override fun visitClass(cls: ClassItem) { 377 // code operating on the class here 378 } 379 }) 380 381Similarly you can visit all items (regardless of type) by overriding 382`visitItem`, or to specifically visit methods, fields and so on overriding 383`visitPackage`, `visitClass`, `visitMethod`, etc. 384 385There is also an `ApiVisitor`. This is a subclass of the `ItemVisitor`, but 386which limits itself to visiting code elements that are part of the API. 387 388This is how for example the SignatureWriter and the StubWriter are both 389implemented: they simply extend `ApiVisitor`, which means they'll only export 390the API items in the codebase, and then in each relevant method they emit the 391signature or stub data: 392 393 class SignatureWriter( 394 private val writer: PrintWriter, 395 private val generateDefaultConstructors: Boolean, 396 private val filter: (Item) -> Boolean) : ApiVisitor( 397 visitConstructorsAsMethods = false) { 398 399 .... 400 401 override fun visitConstructor(constructor: ConstructorItem) { 402 writer.print(" ctor ") 403 writeModifiers(constructor) 404 writer.print(constructor.containingClass().fullName()) 405 writeParameterList(constructor) 406 writeThrowsList(constructor) 407 writer.print(";\n") 408 } 409 410 .... 411 412### Visiting Types 413 414There is a `TypeVisitor` similar to `ItemVisitor` which you can use to visit all 415types in the codebase. 416 417When computing the API, all types that are included in the API should be 418included (e.g. if `List<Foo>` is part of the API then `Foo` must be too). This 419is easy to do with the `TypeVisitor`. 420 421### Diffing Codebases 422 423Another visitor which helps with implementation is the ComparisonVisitor: 424 425 open class ComparisonVisitor { 426 open fun compare(old: Item, new: Item) {} 427 open fun added(item: Item) {} 428 open fun removed(item: Item) {} 429 430 open fun compare(old: PackageItem, new: PackageItem) { } 431 open fun compare(old: ClassItem, new: ClassItem) { } 432 open fun compare(old: MethodItem, new: MethodItem) { } 433 open fun compare(old: FieldItem, new: FieldItem) { } 434 open fun compare(old: ParameterItem, new: ParameterItem) { } 435 436 open fun added(item: PackageItem) { } 437 open fun added(item: ClassItem) { } 438 open fun added(item: MethodItem) { } 439 open fun added(item: FieldItem) { } 440 open fun added(item: ParameterItem) { } 441 442 open fun removed(item: PackageItem) { } 443 open fun removed(item: ClassItem) { } 444 open fun removed(item: MethodItem) { } 445 open fun removed(item: FieldItem) { } 446 open fun removed(item: ParameterItem) { } 447 } 448 449This makes it easy to perform API comparison operations. 450 451For example, metalava has a feature to mark "newly annotated" nullness 452annotations as migrated. To do this, it just extends `ComparisonVisitor`, 453overrides the `compare(old: Item, new: Item)` method, and checks whether the old 454item has no nullness annotations and the new one does, and if so, also marks the 455new annotations as @Migrate. 456 457Similarly, the API Check can simply override 458 459 open fun removed(item: Item) { 460 reporter.report(error, item, "Removing ${Item.describe(item)} is not allowed") 461 } 462 463to flag all API elements that have been removed as invalid (since you cannot 464remove API. (The real check is slightly more complicated; it looks into the 465hierarchy to see if there still is an inherited method with the same signature, 466in which case the deletion is allowed.)) 467 468### Documentation Generation 469 470As mentioned above, metalava generates documentation directly into the stubs 471files, which can then be processed by Dokka and Javadoc to generate the same 472docs as before. 473 474Doclava1 was integrated with javadoc directly, so the way it generated metadata 475docs (such as documenting permissions, ranges and typedefs from annotations) was 476to insert auxiliary tags (`@range`, `@permission`, etc) and then this would get 477converted into English docs later via `macros_override.cs`. 478 479This it not how metalava does it; it generates the English documentation 480directly. This was not just convenient for the implementation (since metalava 481does not use javadoc data structures to pass maps like the arguments for the 482typedef macro), but should also help Dokka -- and arguably the Kotlin code which 483generates the documentation is easier to reason about and to update when it's 484handling loop conditionals. (As a result I for example improved some of the 485grammar, e.g. when it's listing a number of possible constants the conjunction 486is usually "or", but if it's a flag, the sentence begins with "a combination of 487" and then the conjunction at the end should be "and"). 488