Today’s reading is “When Sound Affects Vision: Effects of Auditory Grouping on Visual Motion Perception,” by Katsumi Watanabe and Shinsuke Shimojo of Caltech (Psychological Science, 2001).
In 1997, Sekuler, Sekuler, and Lau discovered a fascinating effect that I’ve attempted to replicate using the crude resources available to me (iMovie). I’ve made two movies (quicktime required) that (perhaps with a little imagination) replicate the effect. First watch movie 1:
It looks like the two balls pass through each other, right?
Now, make sure you have the sound turned up on your computer and watch movie 2.
You should hear a clicking sound, right as the balls meet. Now it looks like the balls “bounce” off each other, right? But the display was identical — the only thing I changed was to add the sound to accompany the “collision.” The simple soundtrack changed your visual perception of the event. Nearly every observer Sekuler et al. showed similar movies to saw it the same way.
Watanabe and Shimojo wanted to know how this effect works. Does any sound at the time of “collision” cause the bouncing effect, or do only certain types of sounds work? So they asked observers to do the same task, but instead of playing a single sound, they played three separate clicks — one before the collision, one at the time of the collision, and one after. Now observers saw the balls pass through each other again.
Next they played a series of tones, but they changed the pitch of the tone simultaneous with the collision. Again observers saw the balls “bouncing” back to their original positions. Watanabe and Shimojo believe this is due to a phenomenon called “auditory grouping” — we don’t hear the series of clicks as separate, disconnected sounds, but as a single group. By changing the pitch of one of the sounds, they cause observers to ungroup the sounds, and this in turn causes the bouncing effect to reappear.
The bouncing effect is yet another example of how our auditory system is linked to the visual system. It’s part of the reason that staged fistfights and swordfights seem “real” — we’re often fooled even by fake punches that “miss” by a foot or more, as long as they’re accompanied by an appropriate sound effect.