Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Submit feedback
Sign in
Toggle navigation
F
Front-End
Project
Project
Details
Activity
Releases
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
abdullh.alsoleman
Front-End
Commits
93c581a7
Unverified
Commit
93c581a7
authored
Dec 16, 2022
by
gaaclarke
Committed by
GitHub
Dec 16, 2022
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Formatted and removed lints from devicelab README.md (#117239)
parent
ebeb4918
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
37 additions
and
28 deletions
+37
-28
README.md
dev/devicelab/README.md
+37
-28
No files found.
dev/devicelab/README.md
View file @
93c581a7
...
...
@@ -7,29 +7,36 @@ the tests are referred to as "tasks" in the API, but since we primarily use it
for testing, this document refers to them as "tests".
Current statuses for the devicelab are available at
https://flutter-dashboard.appspot.com/#/build. See
[
dashboard user guide
](
https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md
)
<https://flutter-dashboard.appspot.com/#/build>
. See
[
dashboard user
guide](https://github.com/flutter/cocoon/blob/master/app_flutter/USER_GUIDE.md)
for information on using the dashboards.
## Table of Contents
*
[
How the DeviceLab runs tests
](
#how-the-devicelab-runs-tests
)
*
[
Running tests locally
](
#running-tests-locally
)
*
[
Writing tests
](
#writing-tests
)
*
[
Adding tests to continuous integration
](
#adding-tests-to-continuous-integration
)
*
[
Adding tests to continuous
integration](#adding-tests-to-continuous-integration)
*
[
Adding tests to presubmit
](
#adding-tests-to-presubmit
)
## How the DeviceLab runs tests
DeviceLab tests are run against physical devices in Flutter's lab (the "DeviceLab").
DeviceLab tests are run against physical devices in Flutter's lab (the
"DeviceLab").
Tasks specify the type of device they are to run on (
`linux_android`
,
`mac_ios`
,
`mac_android`
,
`windows_android`
, etc).
When a device in the lab is free, it will pickup tasks that need to be completed.
Tasks specify the type of device they are to run on (
`linux_android`
,
`mac_ios`
,
`mac_android`
,
`windows_android`
, etc). When a device in the lab is free, it
will pickup tasks that need to be completed.
1.
If the task succeeds, the test runner reports the success and uploads its performance metrics to Flutter's infrastructure. Not
all tasks record performance metrics.
2.
If task fails, an auto rerun happens. Whenever the last run succeeds, the task will be reported as a success. For this case,
a flake will be flagged and populated to the test result.
3.
If the task fails in all reruns, the test runner reports the failure to Flutter's infrastructure and no performance metrics are collected
1.
If the task succeeds, the test runner reports the success and uploads its
performance metrics to Flutter's infrastructure. Not all tasks record
performance metrics.
2.
If task fails, an auto rerun happens. Whenever the last run succeeds, the
task will be reported as a success. For this case, a flake will be flagged and
populated to the test result.
3.
If the task fails in all reruns, the test runner reports the failure to
Flutter's infrastructure and no performance metrics are collected
## Running tests locally
...
...
@@ -63,10 +70,11 @@ To run a test, use option `-t` (`--task`):
Where
`NAME_OR_PATH_OF_TEST`
can be either of:
-
the _name_ of a task, which is a file's basename in
`bin/tasks`
. Example:
`complex_layout__start_up`
.
-
the path to a Dart _file_ corresponding to a task, which resides in
`bin/tasks`
.
Tip: most shells support path auto-completion using the Tab key. Example:
`bin/tasks/complex_layout__start_up.dart`
.
*
the _name_ of a task, which is a file's basename in
`bin/tasks`
. Example:
`complex_layout__start_up`
.
*
the path to a Dart _file_ corresponding to a task, which resides in
`bin/tasks`
. Tip: most shells support path auto-completion using the Tab key.
Example:
`bin/tasks/complex_layout__start_up.dart`
.
To run multiple tests, repeat option
`-t`
(
`--task`
) multiple times:
...
...
@@ -107,19 +115,19 @@ Example:
The
`--ab=10`
tells the runner to run an A/B test 10 times.
`--local-engine=host_debug_unopt`
tells the A/B test to use the
`host_debug_unopt`
engine build.
`--local-engine`
is required for A/B test.
`--local-engine=host_debug_unopt`
tells the A/B test to use the
`host_debug_unopt`
engine build.
`--local-engine`
is required for A/B test.
`--ab-result-file=filename`
can be used to provide an alternate location to
output
the JSON results file (defaults to
`ABresults#.json`
). A single
`#`
character can be
used to indicate where to insert a serial number if a file with that name already
exists, otherwise, the file will be overwritten.
`--ab-result-file=filename`
can be used to provide an alternate location to
output the JSON results file (defaults to
`ABresults#.json`
). A single
`#`
character can be used to indicate where to insert a serial number if a file with
that name already
exists, otherwise, the file will be overwritten.
A/B can run exactly one task. Multiple tasks are not supported.
Example output:
```
```
text
Score Average A (noise) Average B (noise) Speed-up
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 2900.20 (8.44%) 2426.70 (8.94%) 1.20x
bench_card_infinite_scroll.canvaskit.totalUiFrame.average 4964.00 (6.29%) 4098.00 (8.03%) 1.21x
...
...
@@ -142,13 +150,14 @@ Summarize tool example:
ABresults.json ABresults1.json ABresults2.json ...
```
`--[no-]tsv-table`
tells the tool to print the summary in a table with tabs for
easy spreadsheet
entry. (defaults to on)
`--[no-]tsv-table`
tells the tool to print the summary in a table with tabs for
e
asy spreadsheet e
ntry. (defaults to on)
`--[no-]raw-summary`
tells the tool to print all per-run data collected by the
A/B test formatted
with tabs for easy spreadsheet entry. (defaults to on)
`--[no-]raw-summary`
tells the tool to print all per-run data collected by the
A/B test formatted
with tabs for easy spreadsheet entry. (defaults to on)
Multiple trailing filenames can be specified and each such results file will be processed in turn.
Multiple trailing filenames can be specified and each such results file will be
processed in turn.
## Reproducing broken builds locally
...
...
@@ -208,7 +217,7 @@ _TASK_- the name of your test that also matches the name of the
1.
Add target to
[
.ci.yaml
](
https://github.com/flutter/flutter/blob/master/.ci.yaml
)
-
Mirror an existing one that has the recipe
`devicelab_drone`
*
Mirror an existing one that has the recipe
`devicelab_drone`
If your test needs to run on multiple operating systems, create a separate
target for each operating system.
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment