This is neat, but it's often a lot simpler to just render each object relative to the camera instead. Which can be as simple as subtracting the camera's position from the world position right before rendering.