This is a bug in my previous CL: instead of cancelling the timer if
there are no more listeners, I canceled it if there were listeners (I
can claim I just missed a not :) ).

Not cancelling the timer when removing the last listener was not that bad, as
the timer callback is guarded by a check to see if there are listeners.
So the animation will not continue.

But in the case there were multiple listeners on the same stream, and
one of them is removed, this bug will stop the animation for all other
listeners.
I added a test case for this scenario.