This remove a very brittle aspect of flutter drive, whereby it would
assume a known port instead of explicitly finding out what it was.

Fixes #7692 and hopefully fixes the devicelab tests.