We were making local copies of the listener maps, but we were actually
iterating the underlying sets. Now we make local copies of the sets.

Fixes #803