Why study perception? Human Perception and VR • Important to understand human abilities and limits • Determine appropriate range for input and output (frame rate, resolution, etc) • Understand that different senses have different ranges and abilities • Determine appropriate sensory channel to use to transfer information • Avoid sickness and injury Human Perceptual Systems • • • • • Visual Vestibular Auditory Olfactory Haptic Gustatory Visual Perception • Visual information is often considered the primary feature of virtual environments, we gather large amounts of information with vision in everyday life • The visual channel generally can process much larger bandwidth than other senses • This channel is also sensitive to very small anomalies, especially when motion is involved
Tips on Visual Perception • Wider FOV is usually better • Use caution with large amounts of disparity • Use caution with motion, especially when it is fast, or wide FOV is used, or looming is involved • Consider that some simple depth cues may be much easier or cheaper to implement and result in only a slight decline in performance Auditory Perception • Audio in VEs is often absent or implemented poorly (only as an afterthought).
But the entertainment industry recognizes that sound is vital to creating ambience and emotion. • Sound can also be used to transmit useful and important information • Senses such as visual must be directed at the stimulus to receive information – the auditory channel is continually receiving information. • When sound is done poorly, the brain notices! 1 Vestibular System • Senses movements/accelerations of the head/body. • Primary organs used are: Three vestibular canals (rotations) – Otolith organs: Utricle and saccule (gravity and linear accelerations) – These organs are connected Your ear is not just a microphone! Image from www. kyent. com/ hearing_loss_types. htm Vestibular Sense • We generally use information from this sense without conscious thought. We are usually aware of it only when something is wrong. • Vestibular information works together with visual and kinesthetic information to maintain posture.
These senses are also tightly coupled. • This sense is a major factor in motion sickness and cybersickness – people without a working vestibular system do not experience these phenomenon. Olfaction (Smell) • Compared to other senses, olfaction is poorly understood • lack of effective displays and difficulty in producing broad range of stimuli (no “RGB” for smell) make research difficult • available research is highly specific; other research is secretive (funded by perfume companies) Olfaction properties olfactory events (odor sources) may be near or far, but directional sensitivity is generally poor • sensitivity is relatively high – we can detect odors in sparse concentrations (low threshold) • range of sensitivity is large – we can perceive in very small or very large quantities • temporal sensitivity is poor and response times are slow – may need between 20-60 seconds between stimuli to resolve different smells (“cleansing of the palate”) – no real control over stimulus decay rate (without significant air circulation or respiration control) Taste/Gustation • Basic elements of taste Sweet – Sour – Bitter – Salty – Smell – Umami – savory/meaty, Asian origin, found in glutamate Taste receptor distribution (1901 by Deiter Hanig) 2 Taste Issues – High variability between people – Different thresholds for each basic element – humans perceive bitterness in small quantities but need much more salt or sugar to experience a response – Learning/experience – Interactions between taste sensations – Affected by other senses – Strong influence of smell on taste – Need more than flavor, ex: texture, sensations like “fizz” – Primaries seem incomplete: “heat,” astringency, metallic…
Haptics • Greek term, relates to physical contact or touch, includes tactile/cutaneous and kinesthetic/force perception • Separating kinesthetic and tactile perception in humans is nearly impossible; combining them in a computer display is difficult • Haptic sense is bidirectional – touching an object also affects it (unlike seeing or listening) – combination of output and input More Haptics Issues latency in simulation especially noticeable • Unlike visual and audio senses, usually cannot be stimulated from a distance kinesthetic/force feedback require direct physical contact • some cutaneous senses may be remotely stimulated: wind (fans), heat (heat lamps), etc. • capable of multiple simultaneous channels of feedback (different locations at once, or different stimuli on same location) Sensory Comparison Issues • • • • • • • • • relative reaction time/response rate relative acuity or sensitivity directionality, attentiveness stimulus production – can it be remotely induced or does it require direct contact? motional/memory evocations information throughput number of dimensions number of simultaneous/distinguishable channels sensitivity to environmental factors Combining Senses • • • • • • • • • temporal synchronization positional synchronization; collocation relative fidelity dissonance or mismatch Ventriloquist effect – dominance of visual cues (and prior knowledge/experience) over audio localization visual cues have also shown dominance over haptic cues Transference of Object Permanence – making one object in the environment seem very real (e. g. , using haptics) makes the rest of the world seem more real.
Vestibular/visual cues – small/limited motions coupled with proper graphics and visual flow give a strong sense of broader range of physical motion humans are highly adaptable (eg. mapping from horizontal mouse motion to vertical point motion) and learn to overcome mismatches between stimuli – designers should use this capability knowingly and sparingly – should provide re-adaption time where appropriate (e. g. , driving simulation) Level of I M M E R S I O N (in general) VR Displays Desktop Personal VR Benches & Desks Theaters Powerwalls Boom / HMDs Domes Panoramic CAVEs Configurables Enclosures 3 Physical separation
Lenticular (barrier) (images from www. big3d. com) Also see www. artn. com pscholograms (image from http://www. stanford. edu/~matteoja/lent. html) Anaglyph Stereo Passive Stereo (image from Barco http://www. barco. com) (Image from stereo3d. com) Works with CRT and DLP projectors Special screen needed Linear or circular Tricky with LCD Doesn’t work with monitor Inexpensive glasses and filters More ghosting? Active Stereo Alternative VR Displays Volumetric (image from Actuality Systems http://www. actuality-systems. com) Autostereoscopic (image from Stereographics http://www. stereographics. com) (NuVision emitter, image from stereo3d. om) Requires high refresh rate Some are expensive Works with CRT and DLP (monitor or projector) Doesn’t work with LCD (refresh too slow) Retinal (movie from HITLab http://www. hitl. washington. edu/projects/vrd/anim/eye. htm) (image from Stereographics http://www. stereographics. com) 4 Advantages: Desktop Displays (Fishtank VR) Shortcomings: Advantages: Resolution Limited FOV & Visual acuity Limited user motion Brightness Conflicting depth cues Contrast Semi-immersion Stereo-ready? No 3D interaction Familiar interface Small investment Low-maintenance No dedicated space Easy to upgrade Software available image from VRAC, Iowa State University, http://www. vrac. iastate. edu) Blocks out the real world (not always) 1st person interaction Shortcomings: Look-around Low-maintenance Time for in and out Small space User adjustments Movable No tolerant to tech Single viewer Cables Heavy Distortions Visual quality Hygiene (image from Fifth Dimension Technologies Small FOV often http://www. 5dt. com/) HMDs Workbenches/Desks Advantages: Larger FOV Medium investment Relatively low maintenance Sys admins can handle Familiar interaction devices Movable (image from Barco http://www. barco. com) Shortcomings:
Distortion (some) Limited user motion Conflicting depth cues “A little” maintenance Specialized software Potential for reverse stereo (2+ users) (image from Mechdyne/Fakespace http://www. fakespace. com) Advantages: Walls (image from NCSA http://www. ncsa. uiuc. edu/Divisions/ DMV/Vis/Projects/TiledWall/) High Resolution Larger FOV Allows for user motion Shortcomings: Easy in and out High Maintenance Tolerance to tech Specialized software Multiple viewers No look-around Scalable Most non-stereo Space Cost (image from Embraer & Fakespace http://www. fakespace. com) 5 Domes/Theaters/Panoramic Advantages:
Larger FOV Limited look-around Easy in and out Tolerance to tech Multiple viewers Shortcomings: Scalable High Maintenance Specialized software Distortion “sweet” viewing spot Space Cost (image from Elumens http://www. elumens. com/ Advantages: CAVEs (image from USC Institute for Creative Technologies & Fakespace http://www. fakespace. com) Shortcomings: Larger FOV 1st person interaction Look-around Easy in and out Tolerance to tech Multiple viewers Scalable High Maintenance Specialized software (image from Iowa State University, Virtual Reality Applications Center, www. vrac. iastate. edu) Space/non-portable Cost
Advantages: Configurables Shortcomings: More space 2-in-1 Some are more portable and less expensive than CAVE Considerations Image Generators (computers) Speed / Interactivity Price Graphics processing power Single / Double / Quad Buffer Computation power Memory & Texture memory 6 More Considerations Other hardware and software availability and considerations (I/O devices) Synching multiple displays (not just video, but also haptic to video, audio, etc) Upgrade choices Portability / Space requirements Maintenance The Tradition (image from Silicon Graphics Inc. www. sgi. com) An example: SGI Advantages:
Traditionally, excellent graphics abilities Genlock Many processors Large shared memory Fair amount of existing software UNIX-based OS The New Trend Shortcomings: Expensive! Slowed graphics board progress Slower processors Only runs IRIX Big, hot, nonportable, special power and environment (onyx) (image from Dell. www. dell. com) PCs Advantages: Inexpensive Good graphics from gaming boards Windows is “Standard” UNIX-like OS available Portable Other Cluster Solutions • Clusters with special purpose hardware, such as ORAD DVG compositing – – – – Time multiplexing Anti-aliasing Image subdivision “eye” multiplexing
Shortcomings: Only 1 or 2 processors* Windows not always good for VR Small memory (may increase with 64-bit) Only 1 or 2 displays* Genlock hard to come by for multiple displays using a cluster * On a single machine 7 How do we allow users to interact with the virtual world, and how do we make it user-centered? Tracking & Input Devices! Tracking for VR is usually 6 degrees-offreedom (DOF): -3 degrees for location (x,y,z) -3 degrees for orientation (yaw, pitch, roll) Methods/Technologies Magnetic (EM) Mechanical Ultrasound Inertial Optical Image Based Electro Magnetic Emitter Sensors Electro Magnetic image from Ascension Technology www. ascension-tech. com) Mechanical UltraSonic 8 Ultrasound/Inertial HiBall-3000TM Ceiling mounted, infrared LED strips, scalable Smallish sensor (images from Intersense www. isense. com) (images from 3rd Tech www. 3rdtech. com) Input Devices • Hardware that allows the user to communicate with the system • Input device vs. interaction technique • Issues – – – – – – A single device can be used for many different techniques Degrees of freedom Data: discrete vs continuous Number of limbs involved Task to be performed Cost Traditional/Fishtank Devices (image from www. rhino3d. com) User freedom (wires) (image from www. fakespacelabs. com) Novel Keyboard Devices (image from www. evl. uic. edu) Wands (image from www. isense. com) (image from http://halfkeyboard. com) (image from www. wandavr. com) (image from http://canesta. com) (KITTY (Keyboard Independent Touch Typing) image from HITLab) 9 • Pinch Gloves • CyberGloves / 5DT gloves – – – Gloves – Conductive cloth at fingertips and palm. – “pinching” gestures of 2 to 10 fingers, also combinations. Good recognition. – >115,000 gestures – Other devices can be built using the conductive cloth – Relatively inexpensive – Easy on/off (image from www. akespacelabs. com) Senses flex/bend at each joint Uses this to determine postures/gestures “infinite” possibilities Calibration Cost (image from www. vrealities. com) (image from www. immersion. com) Vehicles & Simulators (mockups) Speech/Voice Input • Advantages: – Hands-free – Familiar • Limitations: – Ambiguity possible – Noise and interference – “training” needed, both for system and user Locomotion Devices • • • • Novel Input Devices “Biofeedback” Galvanic skin response Pulse rate Respiration rate (images from www. sarcos. com) (image from www. cs. utah. edu) (image from http://www. media. mit. edu/affect) 0 Haptic Devices • Tactile displays – Give a sense of touch • Vibration • Pneumatic • Electrocutaneous • Force displays – Convey larger scale forces to a user (images from www. microsoft. com) (image from www. immersion. com) (image from http://haptic. mech. nwu. edu/intro/gallery/) (Rutgers Hand Master images from www. caip. rutgers. edu) (images from www. immersion. com) (Phantom by SensAble Tech, image from http://www. stanford. edu/~neel/haptic. scene. representation/photos) Olfactory Displays • Attempts to produce olfactory displays – Scratch-n-Sniff – Sensorama – Smell-O-Vision – Trisenx (www. risenx. com) – ScentAir Technologies – DigiScents – AromaSys (images from www. scentair. com) Universal Kroger • Still searching for the “RGB of scent” 11 (DigiScents iSmell image from www. contourinc. com) AromaSys (www. aromasys. com) Partial client list: • Bellagio • Caesars Palace • MGM Grand • Marriott • Ritz Carlton • Victoria’s Secret Taste Displays • Issues – Intrusive – Safety – Human variability – Learning – Affected by other senses – Need more than flavor, ex: texture (Food Simulator image from www. siggraph. org/s2003) • Basic elements of taste 12