README.md 6.83 KB
Newer Older
1
# Flutter's Build Infrastructure
2

3
This directory exists to support building Flutter on our build infrastructure.
4

5
The results of such builds are viewable at:
6 7
* https://cirrus-ci.com/github/flutter/flutter/master
  - Testing done on PRs and submitted changes on GitHub.
8
* https://build.chromium.org/p/client.flutter/console
9
  - Additional testing and processing done after changes are submitted.
10

11
The external master pages for the Chromium infra bots do not allow
12 13
forcing new builds. Contact @eseidelGoogle or another member of
Google's Flutter team if you need to do that.
14

15 16 17 18 19 20 21
The [Cirrus](https://cirrus-ci.org)-based bots run the [`test.dart`](test.dart)
script for each PR and submission. It does testing for the tools, for the
framework, and (for submitted changes only) rebuilds and updates the master
branch API docs staging site. For tagged dev and beta builds, it also builds and
deploys the gallery app to the app stores.

The rest of this document discusses only the Chromium infra bots.
22 23

This infrastructure is broken into two parts. A buildbot master specified by our
24 25 26 27 28 29
[builders.pyl](https://chromium.googlesource.com/chromium/tools/build.git/+/master/masters/master.client.flutter/builders.pyl)
file, and a [set of
recipes](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter)
which we run on that master.  Both of these technologies are highly specific to
Google's Chromium project. We're just borrowing some of their infrastructure.

30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51
## Prerequisites

- [install depot_tools](http://www.chromium.org/developers/how-tos/install-depot-tools)
- Python package installer: `sudo apt-get install python-pip`
- Python coverage package (only needed for `training_simulation`): `sudo pip install coverage`

## Getting the code

The following will get way more than just recipe code, but it _will_ get the recipe code:

```bash
mkdir chrome_infra
cd chrome_infra
fetch infra
```

More detailed instructions can be found [here](https://chromium.googlesource.com/infra/infra/+/master/doc/source.md).

Most of the functionality for recipes comes from `recipe_modules`, which are
unfortunately spread to many separate repositories.  After checking out the code
search for files named `api.py` or `example.py` under `infra/build`.

52 53 54 55 56
## Editing a recipe

Flutter has one recipe per repository. Currently
[flutter/flutter](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter/flutter.py)
and
Devon Carew's avatar
Devon Carew committed
57 58 59 60
[flutter/engine](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter/engine.py):

- build/scripts/slave/recipes/flutter/flutter.py
- build/scripts/slave/recipes/flutter/engine.py
61

62
Recipes are just Python.  They are
63 64 65
[documented](https://github.com/luci/recipes-py/blob/master/doc/user_guide.md)
by the [luci/recipes-py github project](https://github.com/luci/recipes-py).

66
The typical cycle for editing a recipe is:
Eric Seidel's avatar
Eric Seidel committed
67

68 69
1. Make your edits (probably to files in
   `//chrome_infra/build/scripts/slave/recipes/flutter`).
70 71
2. Run `build/scripts/slave/recipes.py --use-bootstrap test train` to
   update expected files
72
3. Run `build/scripts/slave/recipes.py run flutter/<repo> slavename=<slavename>
73
   mastername=client.flutter buildername=<buildername> buildnumber=1234` where `<repo>` is one
74 75 76 77 78
   of `flutter` or `engine`, and `slavename` and `buildername` can be looked up
   from the *Build Properties* section of a [recent
   build](https://build.chromium.org/p/client.flutter/one_line_per_build).
4. Upload the patch (`git commit`, `git cl upload`) and send it to someone in
   the `recipes/flutter/OWNERS` file for review.
79

80 81 82 83 84 85
## Editing the client.flutter buildbot master

Flutter uses Chromium's fancy
[builders.pyl](https://chromium.googlesource.com/infra/infra/+/master/doc/users/services/buildbot/builders.pyl.md)
master generation system.  Chromium hosts 100s (if not 1000s) of buildbot
masters and thus has lots of infrastructure for turning them up and down.
86 87
Eventually all of buildbot is planned to be replaced by other infrastructure,
but for now flutter has its own client.flutter master.
88 89 90 91 92 93 94 95 96 97 98 99 100

You would need to edit client.flutter's master in order to add slaves (talk to
@eseidelGoogle), add builder groups, or to change the html layout of
https://build.chromium.org/p/client.flutter.  Carefully follow the [builders.pyl
docs](https://chromium.googlesource.com/infra/infra/+/master/doc/users/services/buildbot/builders.pyl.md)
to do so.

## Future Directions

We would like to host our own recipes instead of storing them in
[build](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter).
Support for [cross-repository
recipes](https://github.com/luci/recipes-py/blob/master/doc/cross_repo.md) is
101
in-progress.  If you view the git log of this directory, you'll see we initially
102
tried, but it's not quite ready.
103 104 105

# Android Tools

106 107 108 109 110 111
The Android SDK and NDK used by Flutter's Chrome infra bots are stored in Google Cloud. During the build a bot runs the
`download_android_tools.py` script that downloads the required version of the Android SDK into `dev/bots/android_tools`.

To check which components are currently installed, download the current SDK stored in Google Cloud using the
`download_android_tools.py` script, then `dev/bots/android_tools/sdk/tools/bin/sdkmanager --list`. If you find that some
components need to be updated or installed, follow the steps below:
112 113 114 115 116 117 118

## How to update Android SDK on Google Cloud Storage

1. Run Android SDK Manager and update packages
   `$ dev/bots/android_tools/sdk/tools/android update sdk`
   Use `android.bat` on Windows.

119 120 121 122 123 124 125 126 127 128 129 130
2. Use the UI to choose the packages you want to install and/or update.

3. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --update`. On Windows, run `sdkmanager.bat` instead. If the
   process fails with an error saying that it is unable to move files (Windows makes files and directories read-only
   when another process is holding them open), make a copy of the `dev/bots/android_tools/sdk/tools` directory, run
   the `sdkmanager.bat` from the copy, and use the `--sdk_root` option pointing at `dev/bots/android_tools/sdk`.

4. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --licenses` and accept the licenses for the newly installed
   components. It also helps to run this command a second time and make sure that it prints "All SDK package licenses
   accepted".

5. Run upload_android_tools.py -t sdk
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150
   `$ dev/bots/upload_android_tools.py -t sdk`

## How to update Android NDK on Google Cloud Storage

1. Download a new NDK binary (e.g. android-ndk-r10e-linux-x86_64.bin)
2. cd dev/bots/android_tools
   `$ cd dev/bots/android_tools`

3. Remove the old ndk directory
   `$ rm -rf ndk`

4. Run the new NDK binary file
   `$ ./android-ndk-r10e-linux-x86_64.bin`

5. Rename the extracted directory to ndk
   `$ mv android-ndk-r10e ndk`

6. Run upload_android_tools.py -t ndk
   `$ cd ../..`
   `$ dev/bots/upload_android_tools.py -t ndk`