The buildDirtyElements() method now enforces that we only build deeper
nodes and never go back to build higher nodes during a frame.