README.md 10.1 KB
Newer Older
1
# Flutter's Build Infrastructure
2

3
This directory exists to support building Flutter on our build infrastructure.
4

5
The results of such builds are viewable at:
6 7
* https://cirrus-ci.com/github/flutter/flutter/master
  - Testing done on PRs and submitted changes on GitHub.
8
* https://ci.chromium.org/p/flutter/
9
  - Additional testing and processing done after changes are submitted.
10

11 12 13 14 15 16 17 18 19 20 21
The LUCI infra requires permissions to retrigger or schedule builds. Contact
@kf6gpe or another Google member of the Flutter team if you need to do that.

The [Cirrus](https://cirrus-ci.org)-based bots run the [`test.dart`](test.dart)
script for each PR and submission. This does testing for the tools, for the
framework, and (for submitted changes only) rebuilds and updates the master
branch API docs [staging site](https://master-docs.flutter.dev/).
For tagged dev and beta builds, it also builds and deploys the gallery app to
the app stores. It is configured by the [.cirrus.yml](/.cirrus.yml).

We also have post-commit testing with actual devices, in what we call our
Shi-Hao Hong's avatar
Shi-Hao Hong committed
22
[devicelab](../devicelab/README.md).
23

Chris Bracken's avatar
Chris Bracken committed
24
## LUCI (Layered Universal Continuous Integration)
25 26 27 28 29 30 31 32

A [set of recipes](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter)
are run on Windows, Linux, and Mac machines. The configuration for how many
machines and what kind are managed internally by Google. Contact @kf6gpe or
another Google member of the Flutter team if you suspect changes are needed
there. Both of these technologies are highly specific to the [LUCI](https://github.com/luci)
project, which is the successor to Chromium's infra. We're just borrowing some
of their infrastructure.
33 34

### Prerequisites
35

36
To work on this infrastructure you will need:
37

38
- [depot_tools](https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up)
39 40 41
- Python package installer: `sudo apt-get install python-pip`
- Python coverage package (only needed for `training_simulation`): `sudo pip install coverage`

42 43 44 45 46
To run prepare_package.dart locally:

- Make sure the depot_tools is in your PATH. If you're on Windows, you also need
  an environment variable called DEPOT_TOOLS with the path to depot_tools as value.
- Run `gsutil.py config` (or `python %DEPOT_TOOLS%\gsutil.py` on Windows) to
Dan Field's avatar
Dan Field committed
47
  authenticate with your auth token.
48 49 50 51 52 53 54
- Create a local temp directory. `cd` into it.
- Run `dart [path to your normal Flutter repo]/dev/bots/prepare_package.dart
  --temp_dir=. --revision=[revision to package] --branch=[branch to deploy to]
  --publish`.
- If you're running into gsutil permission issues, check with @Hixie to make sure
  you have the right push permissions.

55
### Getting the code
56

57 58
The following will get way more than just recipe code, but it _will_ get the
recipe code:
59 60 61 62 63 64 65 66 67 68 69 70 71

```bash
mkdir chrome_infra
cd chrome_infra
fetch infra
```

More detailed instructions can be found [here](https://chromium.googlesource.com/infra/infra/+/master/doc/source.md).

Most of the functionality for recipes comes from `recipe_modules`, which are
unfortunately spread to many separate repositories.  After checking out the code
search for files named `api.py` or `example.py` under `infra/build`.

72
### Editing a recipe
73 74 75 76

Flutter has one recipe per repository. Currently
[flutter/flutter](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter/flutter.py)
and
Devon Carew's avatar
Devon Carew committed
77 78 79 80
[flutter/engine](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter/engine.py):

- build/scripts/slave/recipes/flutter/flutter.py
- build/scripts/slave/recipes/flutter/engine.py
81

82
Recipes are just Python with some limitations on what can be imported. They are
83 84 85
[documented](https://github.com/luci/recipes-py/blob/master/doc/user_guide.md)
by the [luci/recipes-py github project](https://github.com/luci/recipes-py).

86
The typical cycle for editing a recipe is:
Eric Seidel's avatar
Eric Seidel committed
87

88 89
1. Make your edits (probably to files in
   `//chrome_infra/build/scripts/slave/recipes/flutter`).
90
2. Update the tests. Run `build/scripts/slave/recipes.py test train` to update
91
   existing expected output to match the new output. Verify completely new test
92 93
   cases by altering the `GenTests` method of the recipe. The recipe is required
   to have 100% test coverage.
Dan Field's avatar
Dan Field committed
94 95 96
3. Run `led get-builder 'luci.flutter.prod:BUILDER_NAME' | led edit -p 'revision="GIT_HASH"' | led edit-recipe-bundle | led launch`, where `BUILDER_NAME` is the builder name (e.g. `Linux Engine`), and 
   `GIT_HASH` is the hash to build (which is important for the engine but not 
   for the framework).
97 98
4. To submit a CL, you need a local branch first (`git checkout -b [some branch name]`).
5. Upload the patch (`git commit`, `git cl upload`) and send it to someone in
99
   the `recipes/flutter/OWNERS` file for review.
100

101
### The infra config repository
102

103 104 105 106 107
The [flutter/infra](https://github.com/flutter/infra) repository contains
configuration files for the dashboard, builder groups, scheduling, and
individual builders. Edits to this may require changes other internal Google
repositories - e.g., to change the operating system or number of machines. If
you want to do that, reach out to @kf6gpe or another member of the Google team.
108

109 110
Each configuration file in that repository has a link in the top comments to a
schema that describes available properties.
111

112
### Future Directions
113 114 115 116 117

We would like to host our own recipes instead of storing them in
[build](https://chromium.googlesource.com/chromium/tools/build.git/+/master/scripts/slave/recipes/flutter).
Support for [cross-repository
recipes](https://github.com/luci/recipes-py/blob/master/doc/cross_repo.md) is
118
in-progress.  If you view the git log of this directory, you'll see we initially
119
tried, but it's not quite ready.
120

121 122

### Android Tools
123

124 125 126
The Android SDK and NDK used by Flutter's Chrome infra bots are stored in Google
Cloud. During the build a bot runs the `download_android_tools.py` script that
downloads the required version of the Android SDK into `dev/bots/android_tools`.
127

128 129 130
To check which components are currently installed, download the current SDK
stored in Google Cloud using the `download_android_tools.py` script, then
`dev/bots/android_tools/sdk/tools/bin/sdkmanager --list`. If you find that some
131
components need to be updated or installed, follow the steps below:
132

133
#### How to update Android SDK on Google Cloud Storage
134 135 136 137 138

1. Run Android SDK Manager and update packages
   `$ dev/bots/android_tools/sdk/tools/android update sdk`
   Use `android.bat` on Windows.

139 140
2. Use the UI to choose the packages you want to install and/or update.

141 142 143 144 145 146 147 148 149 150 151
3. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --update`. On Windows,
   run `sdkmanager.bat` instead. If the process fails with an error saying that
   it is unable to move files (Windows makes files and directories read-only
   when another process is holding them open), make a copy of the
   `dev/bots/android_tools/sdk/tools` directory, run the `sdkmanager.bat` from
   the copy, and use the `--sdk_root` option pointing at
   `dev/bots/android_tools/sdk`.

4. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --licenses` and accept
   the licenses for the newly installed components. It also helps to run this
   command a second time and make sure that it prints "All SDK package licenses
152 153 154
   accepted".

5. Run upload_android_tools.py -t sdk
155 156
   `$ dev/bots/upload_android_tools.py -t sdk`

157
#### How to update Android NDK on Google Cloud Storage
158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174

1. Download a new NDK binary (e.g. android-ndk-r10e-linux-x86_64.bin)
2. cd dev/bots/android_tools
   `$ cd dev/bots/android_tools`

3. Remove the old ndk directory
   `$ rm -rf ndk`

4. Run the new NDK binary file
   `$ ./android-ndk-r10e-linux-x86_64.bin`

5. Rename the extracted directory to ndk
   `$ mv android-ndk-r10e ndk`

6. Run upload_android_tools.py -t ndk
   `$ cd ../..`
   `$ dev/bots/upload_android_tools.py -t ndk`
175

176

177 178
## Flutter codelabs build test

Ian Hickson's avatar
Ian Hickson committed
179 180 181 182 183 184 185 186
The Flutter codelabs exercise Material Components in the form of a
demo application. The code for the codelabs is similar to, but
distinct from, the code for the Shrine demo app in Flutter Gallery.

The Flutter codelabs build test ensures that the final version of the
[Material Components for Flutter
Codelabs](https://github.com/material-components/material-components-flutter-codelabs)
can be built. This test serves as a smoke test for the Flutter
187 188 189 190
framework and should not fail. If it does, please address any issues
in your PR and rerun the test. If you feel that the test failing is
not a direct result of changes made in your PR or that breaking this
test is absolutely necessary, escalate this issue by [submitting an
Ian Hickson's avatar
Ian Hickson committed
191
issue](https://github.com/material-components/material-components-flutter-codelabs/issues/new?title=%5BURGENT%5D%20Flutter%20Framework%20breaking%20PR)
192
to the MDC-Flutter Team.
193 194 195 196 197 198 199 200 201 202 203 204 205 206 207

## Unpublishing published archives

Flutter downloadable archives are built for each release by our continuous
integration systems using the [`prepare_package.dart`](prepare_package.dart)
script, but if something goes very wrong, and a release is published that wasn't
intended to be published, the [`unpublish_package.dart`](unpublish_package.dart)
script may be used to remove the package or packages from the channels in which
they were published.

For example To remove a published package corresponding to the git hash
`d444a455de87a2e40b7f576dc12ffd9ab82fd491`, first do a dry run of the script to
see what it will do:

```
208
$ dart ./unpublish_package.dart --temp_dir=/tmp/foo --revision d444a455de87a2e40b7f576dc12ffd9ab82fd491
209 210 211 212 213 214
```

And once you've verified the output of the dry run to be sure it is what you
want to do, run:

```
215
$ dart ./unpublish_package.dart --confirm --temp_dir=/tmp/foo --revision d444a455de87a2e40b7f576dc12ffd9ab82fd491
216 217 218 219 220 221
```

and it will actually perform the actions. You will of course need to have access
to the cloud storage server and have gsutil installed in order to perform this
operation. Only runs on Linux or macOS systems.

222
See `dart ./unpublish_package.dart --help` for more details.
223 224 225 226 227

Once the package is unpublished, it will not be available from the website for
download, and will not be rebuilt (even though there is a tagged revision in the
repo still) unless someone forces the packaging build to run again at that
revision to rebuild the package.