Rendering Surround Images

You probably know 180° or 360° surround videos that let you look into various directions. The Bino video player can play those videos both in 2D and 3D, and there are some example videos on the Bino website. These videos were rendered with WurblPT, and this article explains how that is done.

A surround camera in a renderer cannot use the traditional view frustum approach since it must be able to look in any direction. Instead, it uses an equirectangular map (lat/lon map) to map any view direction on a unit sphere onto a rectangular image. This example shows a 360° map on the left and a 180° map on the right:

360° maps work like environment maps: when viewed with Bino (or some other suitable program), the viewer can look into any direction. 180° maps cover the half-sphere in front of the viewer.

Side note: equirectangular maps are not optimal for this purpose because they suffer from severe distortions, but everybody uses them anyway; see this paper for square map alternatives, and this paper for cube map alternatives.

As in traditional frustum-based cameras, all rays originate from the camera center. If you want to render stereoscopic 3D images, you traditionally have two camera centers for left and right eye view, and these lie left and right to a common camera center because they both look forward into the scene.

This does not work for surround images anymore, because every view direction requires you to rotate the virtual viewer's head, so there is no single pair of camera centers anymore that would be valid for the whole surround image.

To work around this, an approximation is applied: for each view direction, a new pair of left/right camera centers is generated that is optimal for this direction only. This is equivalent to the approach recommended by Google. This works fine as long as the viewer looks into an approximately horizontal direction, but it has breakdown points when the viewer looks straight up or down. You can notice this if you open the sponza-360-tb.jpg 3D image with Bino and look onto the floor directly beneath you.

Have a look at wurblpt-sponza and wurblpt-rolling-marbles to see how these camera models are used to render the Bino example scenes.