Relighting Framework is a framework for synthetic and real scenes image-based relighting with a fixed viewpoint but with any point / area lights and environment lighting conditions.

The method consists of a large preprocessing stage, in which sets of images with different lighting conditions are provided, and two passes of PCA (Principal Component Analysis) compress data and generate the final matrices for each color component. Those matrices are then used in realtime to calcuate the correct image appeareance given the novel lighting conditions.

The Relighting Framework is based on the Lighting Sensitive Display by Shree Nayar, Peter Belhumeur, and Terry Boult, and extends it to use environment maps.

Images can be synthetic or real photographs. The viewpoint should be fixed. The target object should be in the center of an imaginary cube. Each face of the cube should be divided in a grid of f x f elements. So, those 6 x f x f positions should be used to place one distant point light and take -render- 6 x f x f pictures -images-, i.e., one image for each light position.

First PCA
Consider the collection of n images of a scene over varying lighting. The images may either be synthetic or real. Each image I_i is an image of the scene illuminated by a single distant point light source. Divide the image up into m square blocks each containing p pixels. Let I_i^j denote the jth block in the ith image. For each block in the scene we compute a low-dimensional approximation. We create a p x n matrix I^j as a collection of image blocks in which the ith column of I^j is formed by p pixels from the jth block. Using Singular Value Decomposition (SVD), we find a rank b approximation to I^j as

We represent the illumination as the s vector field, of dimension n x 1. Each element in s is the contribution of the corresponding original distant point light image.

To render an image of the scene as if it were illuminated by s we appeal to the previously calculated matrices E, U, and V. First, we compute a compressed coefficient vector as the product V s. Next, we compute a lighting coefficient vector as the product U V s. Note that the vector U V s has dimensions (m x b) x 1 and that the subvector l^j given by the rows b x (j - 1) + 1 through b x j of U V s contain the coefficients necessary for rendering the jth block. Thus, the subvectors can be unstacked and used to render the jth block of the displayed images as the product E^j l^j. This process is repeated for all m blocks.

Point lights are mapped directly to one element of the s vector.

Area lights are mapped to more than one element of the s vector. If the area light with intensity k occupies e elements, the contribution on each element is k / e.

High-Dynamic Range Cube Map texels are interpolated to avoid problems when the resolution of each face of the cubemap is different from the quantity of images taken from each face of the object. n texels are taken according to the corresponding directions and mapped onto s. Texels contributions are weighted by the solid angle.