We can't read our size in the offset setter because we might be in the middle
of layout. This issue also occured with the sliver-based viewport.