Prof. Taehyun Rhee (Victoria University of Wellington, New Zealand)

Title: Perceptually Based Lighting and Rendering for Mixed Reality on HMDs
Image based lighting (IBL) and differential rendering have been used for lighting and composition in live-action films. In visual effects studio, high-resolution high dynamic range (HDR) panoramic images have been used as the light maps providing real world lighting for photo realistic rendering and composition. To provide a seamless mixture between the virtual and real world objects, intensive computation and additional refinement by skillful artists are often required. The seamlessness is the measure of how perceivably apparent the synthetic object in the final composition is, and how indistinguishable it is from the photographed live action scene. Given that the ultimate receiver of final image is the human eye, the seamlessness of final composition is determined by the Human Visual System (HVS). We conducted psychophysical studies to observe and capture the thresholds of the HVS’s ability to perceive illumination inconsistencies between the rendered subject and the photographed scene. Then, image based lighting (IBL) and illumination composition are perceptually optimized to provide real-time solution for mixed reality (MR). Our method has been implemented into commercial game engines Unity/UE4, and provides seamless lighting for MR rendering suitable for modern head mounted displays (HMDs) requires high frame rates. Although HMDs represent an ideal set-up for immersive viewing for VR/MR content, they have a significant downside; people commonly report visual discomfort when using HMDs. The discomfort symptoms include headaches, nausea, dizziness and eyestrain, comparable to the symptoms commonly reported by sufferers of motion sickness. We are studying the cause
of the visual discomfort and developing a perceptually optimized rendering solution to mitigate it. Our initial implementation realized in a commercial game engine shows approximately 30% reduction of visual discomfort on HMDs.