README.md 9.89 KB
Newer Older
1 2 3 4 5 6 7 8 9 10
# integration_test

This package enables self-driving testing of Flutter code on devices and emulators.
It adapts flutter_test results into a format that is compatible with `flutter drive`
and native Android instrumentation testing.

## Usage

Add a dependency on the `integration_test` and `flutter_test` package in the
`dev_dependencies` section of `pubspec.yaml`. For plugins, do this in the
11 12 13
`pubspec.yaml` of the example app:

```yaml
14 15 16 17 18
dev_dependencies:
  integration_test:
    sdk: flutter
  flutter_test:
    sdk: flutter
19
```
20 21 22 23 24 25 26 27 28

Create a `integration_test/` directory for your package. In this directory,
create a `<name>_test.dart`, using the following as a starting point to make
assertions.

```dart
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';

Dan Field's avatar
Dan Field committed
29 30
void main() {
  IntegrationTestWidgetsFlutterBinding.ensureInitialized();
31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52

  testWidgets("failing test example", (WidgetTester tester) async {
    expect(2 + 2, equals(5));
  });
}
```

### Driver Entrypoint

An accompanying driver script will be needed that can be shared across all
integration tests. Create a file named `integration_test.dart` in the
`test_driver/` directory with the following contents:

```dart
import 'package:integration_test/integration_test_driver.dart';

Future<void> main() => integrationDriver();
```

You can also use different driver scripts to customize the behavior of the app
under test. For example, `FlutterDriver` can also be parameterized with
different [options](https://api.flutter.dev/flutter/flutter_driver/FlutterDriver/connect.html).
53
See the [extended driver](https://github.com/flutter/flutter/blob/master/packages/integration_test/example/test_driver/extended_integration_test.dart) for an example.
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70

### Package Structure

Your package should have a structure that looks like this:

```
lib/
  ...
integration_test/
  foo_test.dart
  bar_test.dart
test/
  # Other unit tests go here.
test_driver/
  integration_test.dart
```

71
[Example](https://github.com/flutter/flutter/tree/master/packages/integration_test/example)
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88

## Using Flutter Driver to Run Tests

These tests can be launched with the `flutter drive` command.

To run the `integration_test/foo_test.dart` test with the
`test_driver/integration_test.dart` driver, use the following command:

```sh
flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/foo_test.dart
```

### Web

Make sure you have [enabled web support](https://flutter.dev/docs/get-started/web#set-up)
89
then [download and run](https://docs.flutter.dev/cookbook/testing/integration/introduction#5b-web)
90 91 92 93 94 95 96 97 98 99 100
the web driver in another process.

Use following command to execute the tests:

```sh
flutter drive \
  --driver=test_driver/integration_test.dart \
  --target=integration_test/foo_test.dart \
  -d web-server
```

101 102 103 104 105
### Screenshots

You can use `integration_test` to take screenshots of the UI rendered on the mobile device or
Web browser at a specific time during the test.

106
This feature is currently supported on Android, iOS, and Web.
107

108
#### Android and iOS
109 110 111 112 113

**integration_test/screenshot_test.dart**

```dart
void main() {
114 115
  final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized()
      as IntegrationTestWidgetsFlutterBinding;
116 117 118 119 120

  testWidgets('screenshot', (WidgetTester tester) async {
    // Build the app.
    app.main();

121
    // This is required prior to taking the screenshot (Android only).
122 123 124 125 126 127 128 129 130 131
    await binding.convertFlutterSurfaceToImage();

    // Trigger a frame.
    await tester.pumpAndSettle();
    await binding.takeScreenshot('screenshot-1');
  });
}
```

You can use a driver script to pull in the screenshot from the device.
132
This way, you can store the images locally on your computer. On iOS, the
133
screenshot will also be available in Xcode test results.
134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158

**test_driver/integration_test.dart**

```dart
import 'dart:io';
import 'package:integration_test/integration_test_driver_extended.dart';

Future<void> main() async {
  await integrationDriver(
    onScreenshot: (String screenshotName, List<int> screenshotBytes) async {
      final File image = File('$screenshotName.png');
      image.writeAsBytesSync(screenshotBytes);
      // Return false if the screenshot is invalid.
      return true;
    },
  );
}
```

#### Web

**integration_test/screenshot_test.dart**

```dart
void main() {
159 160
  final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized()
      as IntegrationTestWidgetsFlutterBinding;
161 162 163 164 165 166 167 168 169 170 171 172

  testWidgets('screenshot', (WidgetTester tester) async {
    // Build the app.
    app.main();

    // Trigger a frame.
    await tester.pumpAndSettle();
    await binding.takeScreenshot('screenshot-1');
  });
}
```

173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210
## Android Device Testing

Create an instrumentation test file in your application's
**android/app/src/androidTest/java/com/example/myapp/** directory (replacing
com, example, and myapp with values from your app's package name). You can name
this test file `MainActivityTest.java` or another name of your choice.

```java
package com.example.myapp;

import androidx.test.rule.ActivityTestRule;
import dev.flutter.plugins.integration_test.FlutterTestRunner;
import org.junit.Rule;
import org.junit.runner.RunWith;

@RunWith(FlutterTestRunner.class)
public class MainActivityTest {
  @Rule
  public ActivityTestRule<MainActivity> rule = new ActivityTestRule<>(MainActivity.class, true, false);
}
```

Update your application's **myapp/android/app/build.gradle** to make sure it
uses androidx's version of `AndroidJUnitRunner` and has androidx libraries as a
dependency.

```gradle
android {
  ...
  defaultConfig {
    ...
    testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
  }
}

dependencies {
    testImplementation 'junit:junit:4.12'

211 212 213
    // https://developer.android.com/jetpack/androidx/releases/test/#1.2.0
    androidTestImplementation 'androidx.test:runner:1.2.0'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'
214 215 216 217 218 219 220 221 222 223
}
```

To run `integration_test/foo_test.dart` on a local Android device (emulated or
physical):

```sh
./gradlew app:connectedAndroidTest -Ptarget=`pwd`/../integration_test/foo_test.dart
```

224 225 226 227 228 229 230
Note:
To use `--dart-define` with `gradlew` you must `base64` encode all parameters,
and pass them to gradle in a comma separated list:
```bash
./gradlew project:task -Pdart-defines="{base64(key=value)},[...]"
```

231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269
## Firebase Test Lab

If this is your first time testing with Firebase Test Lab, you'll need to follow
the guides in the [Firebase test lab
documentation](https://firebase.google.com/docs/test-lab/?gclid=EAIaIQobChMIs5qVwqW25QIV8iCtBh3DrwyUEAAYASAAEgLFU_D_BwE)
to set up a project.

To run a test on Android devices using Firebase Test Lab, use gradle commands to build an
instrumentation test for Android, after creating `androidTest` as suggested in the last section.

```bash
pushd android
# flutter build generates files in android/ for building the app
flutter build apk
./gradlew app:assembleAndroidTest
./gradlew app:assembleDebug -Ptarget=<path_to_test>.dart
popd
```

Upload the build apks Firebase Test Lab, making sure to replace <PATH_TO_KEY_FILE>,
<PROJECT_NAME>, <RESULTS_BUCKET>, and <RESULTS_DIRECTORY> with your values.

```bash
gcloud auth activate-service-account --key-file=<PATH_TO_KEY_FILE>
gcloud --quiet config set project <PROJECT_NAME>
gcloud firebase test android run --type instrumentation \
  --app build/app/outputs/apk/debug/app-debug.apk \
  --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk\
  --timeout 2m \
  --results-bucket=<RESULTS_BUCKET> \
  --results-dir=<RESULTS_DIRECTORY>
```

You can pass additional parameters on the command line, such as the
devices you want to test on. See
[gcloud firebase test android run](https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run).

## iOS Device Testing

270 271
Open `ios/Runner.xcworkspace` in Xcode. Create a test target if you
do not already have one via `File > New > Target...` and select `Unit Testing Bundle`.
272
Change the `Product Name` to `RunnerTests`. Make sure `Target to be Tested` is set to `Runner` and language is set to `Objective-C`.
273
Select `Finish`.
274
Make sure that the **iOS Deployment Target** of `RunnerTests` within the **Build Settings** section is the same as `Runner`.
275 276

Add the new test target to `ios/Podfile` by embedding in the existing `Runner` target.
277

278
```ruby
279
target 'Runner' do
280
  # Do not change existing lines.
281 282
  ...

283 284 285 286
  target 'RunnerTests' do
    inherit! :search_paths
  end
end
287
```
288 289

To build `integration_test/foo_test.dart` from the command line, run:
290

291 292 293
```sh
flutter build ios --config-only integration_test/foo_test.dart
```
294

295
In Xcode, add a test file called `RunnerTests.m` (or any name of your choice) to the new target and
296
replace the file:
297 298

```objective-c
299 300
@import XCTest;
@import integration_test;
301

302
INTEGRATION_TEST_IOS_RUNNER(RunnerTests)
303 304
```

305
Run `Product > Test` to run the integration tests on your selected device.
306 307 308 309 310 311 312

To deploy it to Firebase Test Lab you can follow these steps:

Execute this script at the root of your Flutter app:

```sh
output="../build/ios_integ"
313
product="build/ios_integ/Build/Products"
314 315 316
dev_target="14.3"

# Pass --simulator if building for the simulator.
317
flutter build ios integration_test/foo_test.dart --release
318 319

pushd ios
320 321 322 323 324 325 326
xcodebuild build-for-testing \
  -workspace Runner.xcworkspace \
  -scheme Runner \
  -xcconfig Flutter/Release.xcconfig \
  -configuration Release \
  -derivedDataPath \
  $output -sdk iphoneos
327 328 329 330
popd

pushd $product
zip -r "ios_tests.zip" "Release-iphoneos" "Runner_iphoneos$dev_target-arm64.xctestrun"
331 332 333
popd
```

334
You can verify locally that your tests are successful by running the following command:
335 336

```sh
337 338 339
xcodebuild test-without-building \
  -xctestrun "build/ios_integ/Build/Products/Runner_iphoneos14.3-arm64.xctestrun" \
  -destination id=<YOUR_DEVICE_ID>
340 341 342 343 344
```

Once everything is ok, you can upload the resulting zip to Firebase Test Lab (change the model with your values):

```sh
345 346 347
gcloud firebase test ios run \
  --test "build/ios_integ/Build/Products/ios_tests.zip" \
  --device model=iphone11pro,version=14.1,locale=fr_FR,orientation=portrait
348
```