To run device lab tests against a local engine build, pass the appropriate
To run device lab tests against a local engine build, pass the appropriate
...
@@ -211,31 +195,36 @@ A task runs in its own standalone Dart VM and reports results via Dart VM
...
@@ -211,31 +195,36 @@ A task runs in its own standalone Dart VM and reports results via Dart VM
service protocol. This ensures that tasks do not interfere with each other and
service protocol. This ensures that tasks do not interfere with each other and
lets the CI system time out and clean up tasks that get stuck.
lets the CI system time out and clean up tasks that get stuck.
## Adding tests to the CI environment
## Adding tests to continuous integration
The `manifest.yaml` file describes a subset of tests we run in the CI. To add
your test edit `manifest.yaml` and add the following in the "tasks" dictionary:
```
Host only tests should be added to `flutter_tools`.
{NAME_OF_TEST}:
description: {DESCRIPTION}
stage: {STAGE}
required_agent_capabilities: {CAPABILITIES}
```
Where:
There are several PRs needed to add a DeviceLab task to CI.
-`{NAME_OF_TEST}` is the name of your test that also matches the name of the
_TASK_- the name of your test that also matches the name of the
file in `bin/tasks` without the `.dart` extension.
file in `bin/tasks` without the `.dart` extension.
-`{DESCRIPTION}` is the plain English description of your test that helps
others understand what this test is testing.
1. Add prod builder to [flutter/infra devicelab_config.star](https://github.com/flutter/infra/blob/master/config/devicelab_config.star)
-`{STAGE}` is `devicelab` if you want to run on Android, or `devicelab_ios` if
- Example PR: https://github.com/flutter/infra/pull/401/files
you want to run on iOS.
- This will need to soak for 15 minutes after merged to propagate (should show up in [LUCI console[(https://ci.chromium.org/p/flutter/g/devicelab/console))
-`{CAPABILITIES}` is an array that lists the capabilities required of
- There are various lists for the different testbeds a test can run on
the test agent (the computer that runs the test) to run your test. As of writing,
2. Add task to [flutter/flutter prod_builders.json](https://github.com/flutter/flutter/blob/master/dev/prod_builders.json)
the available capabilities are: `linux`, `linux/android`, `linux-vm`,
- Example PR: https://github.com/flutter/flutter/pull/79913/files
3. After 10 green runs, update [flutter/flutter prod_builders.json](https://github.com/flutter/flutter/blob/master/dev/prod_builders.json) to `flaky:false`
If your test needs to run on multiple operating systems, create a separate test
If your test needs to run on multiple operating systems, create a separate test
for each operating system.
for each operating system.
## Adding tests to presubmit
Flutter's DeviceLab does not currently have capacity to run tests against physical devices in presubmit.
Note that DeviceLab tests should generally require a tethered device. If you are adding host-only tests, considering adding your test to `packages/flutter_tools/test/integration.shard`. Example: https://github.com/flutter/flutter/pull/73577/files"
1. Add try builder to [flutter/infra devicelab_config.star](https://github.com/flutter/infra/blob/master/config/devicelab_config.star)
- Example PR: https://github.com/flutter/infra/pull/401/files
- This will need to soak for 15 minutes after merged to propagate
- There are various lists for the different testbeds a test can run on
2. Add task to [flutter/flutter try_builders.json](https://github.com/flutter/flutter/blob/master/dev/try_builders.json)
- Example PR: https://github.com/flutter/flutter/pull/79913/files
