Figure 1.
Experimental design and hypotheses. A, Experimental design. We used fMRI to measure brain activity while participants read written words and heard subsequent degraded spoken words. Written and spoken words were combined in three conditions: (1) match (identical written/spoken words; e.g., whip–whip), (2) partial mismatch (e.g., kip–pip or pick–pip), and (3) total mismatch (e.g., pit–corn). Participants responded with a button press to indicate whether spoken/written words were “same” or “different” and their confidence. B, Stimulus conditions, responses, and underlying neural mechanisms for representing written/spoken word pairs. In an onset partial mismatch trial (depicted in the third row) the spoken word /pIp/ (“pip”) after written “KIP” can be perceived as “kip” (= “same” response) or “pip” (“different”). This behavioral outcome could be explained by one of two neural mechanisms. First, a representation of common sounds would produce a clear representation of the sounds “ip” (shown in black); this representation would be clearer (black) for trials in which participants report that written/spoken words are the “same” than if participants report that written/spoken words differ. Second, a representation of deviating sounds would produce a clearer representation of the deviating sounds “−k + p” (black) on trials in which participants report that written/spoken words differ and an unclear representation of the deviating sounds “−k + p” (light gray) if participants report that written/spoken words are the “same.” C, Similarity between partial mismatch pairs depends on the underlying neural mechanism. A neural representation of common sounds in written/spoken word pairs predicts that representations for word pairs sharing the same expected sounds (grouped by color, left side) should be more similar (e.g., KIP-/pIp/ share sounds “ip” and are therefore more similar to RIP-/wIp/, also sharing “ip” than to KICK-/pIk/ or RICK-/wIk/ sharing “ick”). In contrast, a neural representation of deviating sounds (i.e., prediction error, grouping by shape, right side), predicts that word pairs sharing the same deviating sounds should be more similar (e.g., KIP-/pIp/ deviate in “−k + p” and should be more similar to KICK-/pIk/, also deviating in “−k + p” than to RIP-/wIp/ and RICK-/wIk/ deviating in “−r + w”). Similar examples apply in other conditions (i.e., offset partial mismatch trials), ensuring that differential representation of onset and offset segments does not favor one or other account.