README.md 4.67 KB
Newer Older
1 2
# Flutter devicelab

Yegor's avatar
Yegor committed
3 4 5 6 7 8 9
"Devicelab" (a.k.a. "cocoon") is a physical lab that tests Flutter on real
Android and iOS devices.

This package contains the code for test framework and the tests. More generally
the tests are referred to as "tasks" in the API, but since we primarily use it
for testing, this document refers to them as "tests".

10 11 12
If you have access to Google's internal network, you can see the continuous
build results from the master branch at <http://go/flutter-dashboard/build.html>.
(There is currently no public view of this data, unfortunately.)
Ian Hickson's avatar
Ian Hickson committed
13 14 15

# Prerequisites

16 17 18
You must set the `ANDROID_HOME` environment variable to run tests on Android. If
you have a local build of the Flutter engine, then you have a copy of the
Android SDK at `.../engine/src/third_party/android_tools/sdk`.
Yegor's avatar
Yegor committed
19 20 21 22 23 24 25 26 27 28 29

# Running tests locally

Do make sure your tests pass locally before deploying to the CI environment.
Below is a handful of commands that run tests in a similar way to how the
CI environment runs them. These commands are also useful when you need to
reproduce a CI test failure locally.

To run a test, use option `-t` (`--task`):

```sh
Ian Hickson's avatar
Ian Hickson committed
30
# from the .../flutter/dev/devicelab directory
31
dart bin/run.dart -t {NAME_OR_PATH_OF_TEST}
Yegor's avatar
Yegor committed
32 33
```

34 35 36 37 38 39 40
Where `NAME_OR_PATH_OF_TEST` can be either of:

- the _name_ of a task, which you can find in the `manifest.yaml` file in this
  directory. Example: `complex_layout__start_up`.
- the path to a Dart _file_ corresponding to a task, which resides in `bin/tasks`.
  Tip: most shells support path auto-completion using the Tab key. Example:
  `bin/tasks/complex_layout__start_up.dart`.
Ian Hickson's avatar
Ian Hickson committed
41

Yegor's avatar
Yegor committed
42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59
To run multiple tests, repeat option `-t` (`--task`) multiple times:

```sh
dart bin/run.dart -t test1 -t test2 -t test3
```

To run all tests defined in `manifest.yaml`, use option `-a` (`--all`):

```sh
dart bin/run.dart -a
```

To run tests from a specific stage, use option `-s` (`--stage`):

```sh
dart bin/run.dart -s {NAME_OF_STAGE}
```

Yegor's avatar
Yegor committed
60
Currently there are only three stages defined, `devicelab`, `devicelab_ios` and `devicelab_win`.
Ian Hickson's avatar
Ian Hickson committed
61

Yegor's avatar
Yegor committed
62 63 64
# Reproducing broken builds locally

If a commit caused a test to fail,
65 66
[the dashboard](http://go/flutter-dashboard/build.html) (requires access to the
Google network, sorry) might look something like this:
Yegor's avatar
Yegor committed
67 68 69 70 71 72 73 74 75 76 77 78 79 80

![Broken Test](images/broken-test.png)

The red circle tells you that a test failed. The number inside tells you how
many times the devicelab attempted to run the test before giving up on it.

To reproduce the breakage locally `git checkout` the corresponding Flutter
revision. Note the name of the test that failed. In the example above the
failing test is `flutter_gallery__transition_perf`. This name can be passed to
the `run.dart` command. For example:

```sh
dart bin/run.dart -t flutter_gallery__transition_perf
```
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140

# Writing tests

A test is a simple Dart program that lives under `bin/tests` and uses
`package:flutter_devicelab/framework/framework.dart` to define and run a _task_.

Example:

```dart
import 'dart:async';

import 'package:flutter_devicelab/framework/framework.dart';

Future<Null> main() async {
  await task(() async {
    ... do something interesting ...

    // Aggregate results into a JSONable Map structure.
    Map<String, dynamic> testResults = ...;

    // Report success.
    return new TaskResult.success(testResults);

    // Or you can also report a failure.
    return new TaskResult.failure('Something went wrong!');
  });
}
```

Only one `task` is permitted per program. However, that task can run any number
of tests internally. A task has a name. It succeeds and fails independently of
other tasks, and is reported to the dashboard independently of other tasks.

A task runs in its own standalone Dart VM and reports results via Dart VM
service protocol. This ensures that tasks do not interfere with each other and
lets the CI system time out and clean up tasks that get stuck.

# Adding tests to the CI environment

The `manifest.yaml` file describes a subset of tests we run in the CI. To add
your test edit `manifest.yaml` and add the following in the "tasks" dictionary:

```
  {NAME_OF_TEST}:
    description: {DESCRIPTION}
    stage: {STAGE}
    required_agent_capabilities: {CAPABILITIES}
```

Where:

 - `{NAME_OF_TEST}` is the name of your test that also matches the name of the
 file in `bin/tests` without the `.dart` extension.
 - `{DESCRIPTION}` is the plain English description of your test that helps
 others understand what this test is testing.
 - `{STAGE}` is `devicelab` if you want to run on Android, or `devicelab_ios` if
 you want to run on iOS.
 - `{CAPABILITIES}` is an array that lists the capabilities required of
 the test agent (the computer that runs the test) to run your test. Available
 capabilities are: `has-android-device`, `has-ios-device`.