Unverified Commit 781cd4d8 authored by Casey Hillers's avatar Casey Hillers Committed by GitHub

Update Devicelab README with new dashboard information (#48908)

parent cad08149
...@@ -1023,18 +1023,6 @@ final Set<Hash256> _grandfatheredBinaries = <Hash256>{ ...@@ -1023,18 +1023,6 @@ final Set<Hash256> _grandfatheredBinaries = <Hash256>{
Hash256(0x4C67221E25626CB2, 0x3F94E1F49D34E4CF, 0x3A9787A514924FC5, 0x9EF1E143E5BC5690), Hash256(0x4C67221E25626CB2, 0x3F94E1F49D34E4CF, 0x3A9787A514924FC5, 0x9EF1E143E5BC5690),
// HISTORICAL DEVICE LAB IMAGES
// dev/devicelab/images/agent-statuses.png
Hash256(0x0A43016C84095771, 0x1C610E1C01B1C3B2, 0x681DA0B2062C02AA, 0x31CC7FB3FDC298FD),
// dev/devicelab/images/broken-test.png
Hash256(0x319459F42967A888, 0x90B20063544D6707, 0x849E1E3447CC56A5, 0xE226C47DE34F13AD),
// dev/devicelab/images/legend.png
Hash256(0x92A98975AF5F076C, 0xE7BFAB86B0DD7A3D, 0xC45287B706D10456, 0x4E512BA3C41B2749),
// MISCELLANEOUS // MISCELLANEOUS
// dev/bots/serviceaccount.enc // dev/bots/serviceaccount.enc
...@@ -1060,7 +1048,7 @@ Future<void> verifyNoBinaries(String workingDirectory, { Set<Hash256> grandfathe ...@@ -1060,7 +1048,7 @@ Future<void> verifyNoBinaries(String workingDirectory, { Set<Hash256> grandfathe
assert( assert(
_grandfatheredBinaries _grandfatheredBinaries
.expand<int>((Hash256 hash) => <int>[hash.a, hash.b, hash.c, hash.d]) .expand<int>((Hash256 hash) => <int>[hash.a, hash.b, hash.c, hash.d])
.reduce((int value, int element) => value ^ element) == 0x17138C817203D1A7 // Please do not modify this line. .reduce((int value, int element) => value ^ element) == 0x606B51C908B40BFA // Please do not modify this line.
); );
grandfatheredBinaries ??= _grandfatheredBinaries; grandfatheredBinaries ??= _grandfatheredBinaries;
if (!Platform.isWindows) { // TODO(ianh): Port this to Windows if (!Platform.isWindows) { // TODO(ianh): Port this to Windows
......
# Flutter devicelab # Flutter devicelab
"Devicelab" (a.k.a. "cocoon") is a physical lab that tests Flutter on real "Devicelab" (a.k.a. [Cocoon](https://github.com/flutter/cocoon)) is a physical
Android and iOS devices. lab that tests Flutter on real Android and iOS devices.
This package contains the code for test framework and the tests. More generally This package contains the code for test framework and the tests. More generally
the tests are referred to as "tasks" in the API, but since we primarily use it the tests are referred to as "tasks" in the API, but since we primarily use it
for testing, this document refers to them as "tests". for testing, this document refers to them as "tests".
Build results are available at https://flutter-dashboard.appspot.com. Current statuses for the devicelab are available at
https://flutter-dashboard.appspot.com.
# Reading the dashboard # Dashboards
## The build page ## Build dashboard
The build page is accessible at https://flutter-dashboard.appspot.com/build.html. The build page is accessible at https://flutter-dashboard.appspot.com/#/build.
This page reports the health of build servers, called _agents_, and the statuses This page reports the build statuses of commits to the flutter/flutter repo.
of build tasks.
### Agents
A green agent is considered healthy and ready to receive new tasks to build. A
red agent is broken and does not receive new tasks.
In the example below, the dashboard shows that the `linux2` agent is broken and
requires attention. All other agents are healthy.
![Agent statuses](images/agent-statuses.png)
### Tasks ### Tasks
The table below the agent statuses displays the statuses of build tasks. Task Task statuses are color-coded in the following manner:
statuses are color-coded. The following statuses are available:
**New task** (light blue): the task is waiting for an agent to pick it up and **New task** (blue): the task is waiting for an agent to pick it up and
start the build. start the build.
**Task is running** (spinning blue): an agent is currently building the task. **Task is running** (blue with clock): an agent is currently building the task.
**Task succeeded** (green): an agent reported a successful completion of the **Task succeeded** (green): an agent reported a successful completion of the
task. task.
...@@ -45,13 +34,11 @@ latest attempt succeeded (we currently only try twice). ...@@ -45,13 +34,11 @@ latest attempt succeeded (we currently only try twice).
**Task failed** (red): the task failed all of the attempts. **Task failed** (red): the task failed all of the attempts.
**Task underperformed** (orange): currently not used. **Task is rerunning** (orange): the task is being rerun.
**Task was skipped** (transparent): the task is not scheduled for a build. This **Task was skipped** (transparent): the task is not scheduled for a build. This
usually happens when a task is removed from `manifest.yaml` file. usually happens when a task is removed from `manifest.yaml` file.
**Task status unknown** (purple): currently not used.
In addition to color-coding, a task may display a question mark. This means In addition to color-coding, a task may display a question mark. This means
that the task was marked as flaky manually. The status of such task is ignored that the task was marked as flaky manually. The status of such task is ignored
when considering whether the build is broken or not. For example, if a flaky when considering whether the build is broken or not. For example, if a flaky
...@@ -59,18 +46,11 @@ task fails, GitHub will not prevent PR submissions. However, if the latest ...@@ -59,18 +46,11 @@ task fails, GitHub will not prevent PR submissions. However, if the latest
status of a non-flaky task is red, all pending PRs will contain a warning about status of a non-flaky task is red, all pending PRs will contain a warning about
the broken build and recommend caution when submitting. the broken build and recommend caution when submitting.
Legend: Clicking a cell will pop up an overlay with information about that task. It
includes information such as the task name, number of attempts, run time,
![Task status legend](images/legend.png) queue time, whether it is manually marked flaky, and the agent it was run on.
It has actions to download the log, rerun the task, and view the agent on
The example below shows that commit `e122d5d` caused a wide-spread breakage, the agent dashboard.
which was fixed by `bdc6f10`. It also shows that Cirrus and Chrome
Infra (left-most tasks) decided to skip building these commits. Hovering over
a cell will pop up a tooltip containing the name of the broken task. Clicking
on the cell will open the log file in a new browser tab (only visible to core
contributors as of today).
![Broken Test](images/broken-test.png)
## Why is a task stuck on "new task" status? ## Why is a task stuck on "new task" status?
...@@ -81,7 +61,19 @@ may skip some commits. For example, Cirrus will only test the ...@@ -81,7 +61,19 @@ may skip some commits. For example, Cirrus will only test the
_last_ commit of a PR that's merged into the `master` branch. Chrome Infra may _last_ commit of a PR that's merged into the `master` branch. Chrome Infra may
skip commits when they come in too fast. skip commits when they come in too fast.
## How the devicelab runs the tasks ## Agent dashboard
Agent statuses are available at https://flutter-dashboard.appspot.com/#/agents.
A green agent is considered healthy and ready to receive new tasks to build. A
red agent is broken and does not receive new tasks.
## Performance dashboard
Flutter benchmarks are available at
https://flutter-dashboard.appspot.com/benchmarks.html.
# How the devicelab runs tasks
The devicelab agents have a small script installed on them that continuously The devicelab agents have a small script installed on them that continuously
asks the CI server for tasks to run. When the server finds a suitable task for asks the CI server for tasks to run. When the server finds a suitable task for
...@@ -238,5 +230,10 @@ Where: ...@@ -238,5 +230,10 @@ Where:
- `{STAGE}` is `devicelab` if you want to run on Android, or `devicelab_ios` if - `{STAGE}` is `devicelab` if you want to run on Android, or `devicelab_ios` if
you want to run on iOS. you want to run on iOS.
- `{CAPABILITIES}` is an array that lists the capabilities required of - `{CAPABILITIES}` is an array that lists the capabilities required of
the test agent (the computer that runs the test) to run your test. Available the test agent (the computer that runs the test) to run your test. As of writing,
capabilities are: `has-android-device`, `has-ios-device`. the available capabilities are: `linux`, `linux/android`, `linux-vm`,
`mac`, `mac/ios`, `mac/iphonexs`, `mac/ios32`, `mac-catalina/ios`,
`mac-catalina/android`, `ios/gl-render-image`, `windows`, `windows/android`.
If your test needs to run on multiple operating systems, create a separate test
for each operating system.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
This diff was suppressed by a .gitattributes entry.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment