Rendering with OpenCV Camera Intrinsics and Lens Distortion

In the basic pinhole camera model in Computer Graphics, you get images that are perfectly in focus and do not suffer from distortions. In contrast, real cameras have lens systems that cause depth-of-field and lens distortion effects (among others).

Renderers typically give you a camera parameter to control depth-of-field, and they might give you some way to approximate lens distortion effects. Alternatively, lens distortion can be approximated in a postprocessing step.

WurblPT goes one step further. You can define the camera optics directly from OpenCV camera model parameters: width, height, center pixel and focal lengths determine the view frustum, and distortion parameters k1, k2, k3, p1, p2 determine the lens distortion.

This lets you render images as if they were recorded with a camera that you just calibrated using OpenCV, which is useful for camera simulation and Ground Truth generation for training and validation purposes.

Examples for lens distortions: no distortion (top left), barrel distortion (k < 0, top right), pincushion distortion (k > 0, bottom left), tangential distortion (p ≠ 0, bottom right)
Image without distortion Image with barrel distortion
Image with pinchushion distortion Image with tangential distortion

The OpenCV camera model describes the mapping from an undistorted image to a distorted image. To generate view rays for rendering, we need the inverse: from distorted image coordinates (because we want to generate a distorted image) to undistorted image coordinates (so that we can apply the camera frustum to generate a view direction).

There is no closed-form inverse of the general OpenCV camera model; WurblPT applies an iterative process that works well for a wide range of cameras and supports all parameters of the model.

In special cases, you can choose a restricted closed-from inverse that supports only a subset of the parameters with limited value ranges. See the following papers for details on these special cases: