This pattern breaks when using compositing because we need to lift those
operations into the compositing tree. This patch removes all the ones I could
find and adds an assert to help prevent more from getting introduced.

Fixes #191