Driver training using fused reality

This paper describes a novel mixed reality technique for real-time, color-based video processing for training applications combining software and off-the-shelf hardware called, “fused reality”. This technique allows an operator to view and interact with the physical environment in real-time while viewing the virtual environment through color-designated portals (e.g. painted window panels). Additionally, physical objects can be deployed real-time into the virtual scene (e.g., users can interact with real people embedded within the virtual environment). Fused reality’s chroma keying process allows for realistic set lighting, colors, and user movement and positioning. It also enables multiple keying colors to be used (versus just blue or green). Examples of training applications using fused reality will be provided for parachute training and driving simulation.