README.md 8.77 KB
Newer Older
1
# Flutter's Build Infrastructure
2

3
This directory exists to support building Flutter on our build infrastructure.
4

5 6
Flutter build results are available at:
* https://flutter-dashboard.appspot.com/#/build
Shi-Hao Hong's avatar
Shi-Hao Hong committed
7
  - Aggregate dashboard of the separate CI systems used by Flutter.
8
* https://cirrus-ci.com/github/flutter/flutter/master
9
  - Testing is done on PRs and submitted changes on GitHub.
10

11 12 13 14
Flutter infra requires special permissions to retrigger builds on the
[build dashboard](https://flutter-dashboard.appspot.com/#/build). File an
[infra ticket](https://github.com/flutter/flutter/wiki/Infra-Ticket-Queue) to
request permission.
15 16 17 18 19 20 21 22

The [Cirrus](https://cirrus-ci.org)-based bots run the [`test.dart`](test.dart)
script for each PR and submission. This does testing for the tools, for the
framework, and (for submitted changes only) rebuilds and updates the master
branch API docs [staging site](https://master-docs.flutter.dev/).
For tagged dev and beta builds, it also builds and deploys the gallery app to
the app stores. It is configured by the [.cirrus.yml](/.cirrus.yml).

23 24
The build dashboard includes post-commit testing run on physical devices. See
[//dev/devicelab](../devicelab/README.md) for more information.
25

Chris Bracken's avatar
Chris Bracken committed
26
## LUCI (Layered Universal Continuous Integration)
27

28 29
A [set of infra scripts](https://flutter.googlesource.com/recipes/)
run on Windows, Linux, and Mac machines. The configuration for how many
30 31
machines and what kind are managed internally by Google. File an
[infra ticket](https://github.com/flutter/flutter/wiki/Infra-Ticket-Queue)
32
to request new machine types to be added. Both of these technologies are highly
33 34
specific to the [LUCI](https://github.com/luci) project, which is the successor
to Chromium's infra and the foundation to Flutter's infrastructure.
35 36

### Prerequisites
37

38
To work on this infrastructure you will need:
39

40
- [depot_tools](https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/depot_tools_tutorial.html#_setting_up)
41 42 43
- Python package installer: `sudo apt-get install python-pip`
- Python coverage package (only needed for `training_simulation`): `sudo pip install coverage`

44
To run `prepare_package.dart` locally:
45

46 47
- Make sure the `depot_tools` is in your `PATH`. If you're on Windows, you also need
  an environment variable called `DEPOT_TOOLS` with the path to `depot_tools` as value.
48
- Run `gsutil.py config` (or `python3 %DEPOT_TOOLS%\gsutil.py` on Windows) to
Dan Field's avatar
Dan Field committed
49
  authenticate with your auth token.
50 51 52 53
- Create a local temp directory. `cd` into it.
- Run `dart [path to your normal Flutter repo]/dev/bots/prepare_package.dart
  --temp_dir=. --revision=[revision to package] --branch=[branch to deploy to]
  --publish`.
54
- If you're running into `gsutil` permission issues, check with @Hixie to make sure
55 56
  you have the right push permissions.

57
### Editing a recipe
58

59
Flutter has several recipes depending on the test. The recipes share common
60
actions through `recipe_modules`. Searching the builder config in [infra](https://flutter.googlesource.com/infra/+/refs/heads/main)
61
will indicate the recipe used for a test.
62

63
Recipes are just Python with some limitations on what can be imported. They are
64
[documented](https://github.com/luci/recipes-py/blob/master/doc/user_guide.md)
65
by the [luci/recipes-py GitHub project](https://github.com/luci/recipes-py).
66

67
The typical cycle for editing a recipe is:
Eric Seidel's avatar
Eric Seidel committed
68

69
1. Check out the recipes project using `git clone https://flutter.googlesource.com/recipes`.
70 71 72
2. Make your edits (probably to files in
   `//recipes/recipes`).
3. Update the tests. Run `recipes.py test train` to update
73
   the existing expected output to match the new output. Verify completely new test
74 75
   cases by altering the `GenTests` method of the recipe. The recipe is required
   to have 100% test coverage.
keyonghan's avatar
keyonghan committed
76 77
4. Run `led get-builder 'luci.flutter.staging:BUILDER_NAME' | led edit -pa git_ref='refs/pull/<PR number>/head' | led edit -pa git_url='https://github.com/flutter/<repo>' | led edit-recipe-bundle | led launch`, where `BUILDER_NAME` is the builder name (e.g. `Linux Engine`), and
   `git_ref`/`git_url` is the ref/url of the intended changes to build.
78
   * If `led` fails, ensure that your `depot_tools` checkout is up to date.
79
5. To submit a CL, you need a local branch first (`git checkout -b [some branch name]`).
80 81 82
6. Upload the patch (`git commit`, `git cl upload`), and open the outputted URL to the CL.
7. Use "Find owners" to get reviewers for the CL

83
### Android Tools
84

85
The Android SDK and NDK used by Flutter's Chrome infra bots are stored in Google
86
Cloud. During the build, a bot runs the `download_android_tools.py` script that
87
downloads the required version of the Android SDK into `dev/bots/android_tools`.
88

89 90 91
To check which components are currently installed, download the current SDK
stored in Google Cloud using the `download_android_tools.py` script, then
`dev/bots/android_tools/sdk/tools/bin/sdkmanager --list`. If you find that some
92
components need to be updated or installed, follow the steps below:
93

94
#### How to update Android SDK on Google Cloud Storage
95 96 97 98 99

1. Run Android SDK Manager and update packages
   `$ dev/bots/android_tools/sdk/tools/android update sdk`
   Use `android.bat` on Windows.

100 101
2. Use the UI to choose the packages you want to install and/or update.

102 103 104 105 106 107 108 109 110 111 112
3. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --update`. On Windows,
   run `sdkmanager.bat` instead. If the process fails with an error saying that
   it is unable to move files (Windows makes files and directories read-only
   when another process is holding them open), make a copy of the
   `dev/bots/android_tools/sdk/tools` directory, run the `sdkmanager.bat` from
   the copy, and use the `--sdk_root` option pointing at
   `dev/bots/android_tools/sdk`.

4. Run `dev/bots/android_tools/sdk/tools/bin/sdkmanager --licenses` and accept
   the licenses for the newly installed components. It also helps to run this
   command a second time and make sure that it prints "All SDK package licenses
113 114 115
   accepted".

5. Run upload_android_tools.py -t sdk
116 117
   `$ dev/bots/upload_android_tools.py -t sdk`

118
#### How to update Android NDK on Google Cloud Storage
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135

1. Download a new NDK binary (e.g. android-ndk-r10e-linux-x86_64.bin)
2. cd dev/bots/android_tools
   `$ cd dev/bots/android_tools`

3. Remove the old ndk directory
   `$ rm -rf ndk`

4. Run the new NDK binary file
   `$ ./android-ndk-r10e-linux-x86_64.bin`

5. Rename the extracted directory to ndk
   `$ mv android-ndk-r10e ndk`

6. Run upload_android_tools.py -t ndk
   `$ cd ../..`
   `$ dev/bots/upload_android_tools.py -t ndk`
136

137

138 139
## Flutter codelabs build test

Ian Hickson's avatar
Ian Hickson committed
140 141 142 143 144 145 146 147
The Flutter codelabs exercise Material Components in the form of a
demo application. The code for the codelabs is similar to, but
distinct from, the code for the Shrine demo app in Flutter Gallery.

The Flutter codelabs build test ensures that the final version of the
[Material Components for Flutter
Codelabs](https://github.com/material-components/material-components-flutter-codelabs)
can be built. This test serves as a smoke test for the Flutter
148 149 150 151
framework and should not fail. If it does, please address any issues
in your PR and rerun the test. If you feel that the test failing is
not a direct result of changes made in your PR or that breaking this
test is absolutely necessary, escalate this issue by [submitting an
Ian Hickson's avatar
Ian Hickson committed
152
issue](https://github.com/material-components/material-components-flutter-codelabs/issues/new?title=%5BURGENT%5D%20Flutter%20Framework%20breaking%20PR)
153
to the MDC-Flutter Team.
154 155 156 157 158 159 160 161 162 163 164 165 166 167 168

## Unpublishing published archives

Flutter downloadable archives are built for each release by our continuous
integration systems using the [`prepare_package.dart`](prepare_package.dart)
script, but if something goes very wrong, and a release is published that wasn't
intended to be published, the [`unpublish_package.dart`](unpublish_package.dart)
script may be used to remove the package or packages from the channels in which
they were published.

For example To remove a published package corresponding to the git hash
`d444a455de87a2e40b7f576dc12ffd9ab82fd491`, first do a dry run of the script to
see what it will do:

```
169
$ dart ./unpublish_package.dart --temp_dir=/tmp/foo --revision d444a455de87a2e40b7f576dc12ffd9ab82fd491
170 171 172 173 174 175
```

And once you've verified the output of the dry run to be sure it is what you
want to do, run:

```
176
$ dart ./unpublish_package.dart --confirm --temp_dir=/tmp/foo --revision d444a455de87a2e40b7f576dc12ffd9ab82fd491
177 178
```

179 180
and it will perform the actions. You will of course need to have access
to the cloud storage server and have `gsutil` installed to perform this
181 182
operation. Only runs on Linux or macOS systems.

183
See `dart ./unpublish_package.dart --help` for more details.
184 185 186 187 188

Once the package is unpublished, it will not be available from the website for
download, and will not be rebuilt (even though there is a tagged revision in the
repo still) unless someone forces the packaging build to run again at that
revision to rebuild the package.