Decoupling speech processing from time

Speech: it’s about time

Time is at the heart of language. A sentence’s meaning is grounded in word order (in many languages), and words are defined by the sequence of their constituents (e.g., peel vs. leap). Moreover, speech is transient, requiring listeners to accumulate information over time and navigate moments of ambiguity when there is insufficient information to recognize a word. These demands are compounded by the rapid pace of speech. Thus, time is a fundamental issue facing the science of speech comprehension.

In response, research in cognitive science has emphasized theories that explain how speech understanding unfolds over time [1–3] and methods that characterize this process, including electrophysiology [4–6] (see Glossary), eye-tracking during reading [7], and the Visual World Paradigm [8, 9]. Research in spoken word recognition and speech perception has converged on a set of guiding principles, which we term the Strong Temporally Coupled Framework (STCF). Broadly speaking, the STCF suggests that language processing is strongly coupled to the unfolding of speech input and that temporal order is of fundamental importance [c.f., 3]. All models of spoken word recognition [10–18], including models of multilingual processing [19, 20], are shaped by these principles.

However, converging evidence suggests that the dynamics of word recognition are only loosely coupled to the unfolding input, and in some cases, they are decoupled entirely. This evidence comes from distinct subdomains of speech perception and word recognition, yet their collective implication for the STCF has not been widely considered (though see [21, 22]). This manuscript synthesizes this work. We first outline the STCF before synthesizing individual lines of research that challenge it. Existing models are likely unable to fully address these issues. Thus, we conclude by proposing speculative first steps toward a solution and a call for novel frameworks to account for the timecourse of speech processing.

The Strong Temporally Coupled Framework

Pioneering research on word recognition emphasized the issue of temporal ambiguity [c.f., 23, for a historical review]. The initial sound of teacher (/ti/) is consistent with teacher, team, and teeth. Listeners solve this problem by immediately activating multiple lexical candidates. As information accumulates, activation incrementally reflects each candidate’s ongoing match to the input (/...ʧɝ/ continues to activate teacher, but not team; [24]; Box 1).

This description is ubiquitous. It appears in hundreds of empirical articles (e.g., [25–27]), including many of the current authors’ (e.g., [28]). Broadly, it assumes three principles (see [3] for these same principles applied to language more broadly):

1. Immediate and Incremental Processing. Speech input is immediately integrated into higher-level representations (e.g., candidate words), which are incrementally updated by subsequent input.

2. Rapid Decay. Information decays rapidly to facilitate incremental updating. Representations of speech input are mapped to words and decay to make way for the next input. Similarly, words are integrated into a sentence-level model and decay to avoid interfering with the next word.

3. Ordered Lexical Representations. Lexical representations are structured around the order of their constituents. A candidate’s activation reflects its match between this internal temporal structure and the order of input (/ti/ activates team, not meat).

Decades of evidence support immediacy and incrementality [e.g., 24, 29–31]. Less work has directly examined decay and ordered representations, but these assumptions are common in computational models [11, 15], which often cannot function without them. This situates the STCF less as an explicitly articulated theory to be tested, and more as a guiding framework, a first-order explanation of psycholinguistic phenomena and models. While researchers in the field can likely point to empirical findings that challenge these principles individually, our goal here is to synthesize this body of evidence. We believe this is important because the STCF has remained dominant in guiding the development of new theories, models, and experiments.

This is evidenced by the fact that these assumptions appear explicitly in textbooks [32], handbooks [33, 34], and review articles [35–37]. Immediate and incremental processing has been described as a point of consensus among researchers [38]. The necessity of including decay mechanisms in models of word recognition has been emphasized [39]. And word similarity judgments are modeled as a function of sound overlap at corresponding temporal “slots” [40, 41]. Both immediacy and rapid decay are reflected in the “categorize and discard” framework discussed in speech perception [42, 43] and the “chunk and pass” principle proposed in language more broadly [3], both of which are proposed to balance the constraints of efficiency against available cognitive resources.

Moreover, every notable model of spoken word recognition is trellised by these assumptions. The neighborhood activation model [16] assumes that similarity between words is dictated by temporal order—leaf and feel are not neighbors, despite sharing the same sounds. In exemplar models, lexical representations are ordered snippets of fine-grained acoustic detail [13], and Bayesian models update the likelihood of a word incrementally and immediately relative to a temporally ordered representation [18]. Distributed recurrent models also assume strong incrementality and temporally ordered representations [12, 14]. Interactive activation models [11, 15, 17, 44], by far the most common approach in this domain, explicitly describe information as flowing immediately and incrementally across levels before rapidly decaying, and they typically feature implicitly ordered lexical representations.

As we describe, some models might (in principle) offer sufficient computational flexibility to capture other approaches to time. However, this (in)ability misses the point. Models reflect theoretical commitments and capture how the field conceptualizes underlying processes. The fact that models are developed to reflect the STCF (at least in part) and are tested on whether they capture these phenomena suggests the field’s theoretical understanding continues to reflect its principles.

Over the decades, the STCF has evolved to adopt more flexibility than its initial conception allowed. Few models since the Cohort model [45] have implemented the STCF with discrete decisions. Instead, word recognition is conceptualized as gradient, allowing for revision of prior commitments [17] and consideration of candidates that partially mismatch the input (rhymes and embedded words [29, 46]). However, candidates are activated immediately, processing remains incremental, and activation is relative to a word’s temporal match with the input (Figure IA). The result is that rhyme candidates are activated later and compete less strongly than onset candidates (Figure IB)—reflecting their temporal overlap with the input. Thus, these effects have not led to a fundamental reconsideration of the STCF.

Similarly, current enthusiasm around prediction [47–49] is consistent with the STCF. In speech perception and word recognition, most predictive processes are grounded in acoustic cues (like coarticulation) that precede a word (e.g., [50, 51]). Use of these cues to predict the next word accelerates the process of activating candidates, enhancing the instantiation of immediacy, and reinforcing temporally structured representations (see [52]).

However, there is a long-known phenomenon that is not well explained by STCF principles. Early work in speech perception found that phonemic boundaries can be shifted by later arriving information (e.g., a consonant judgement can be modulated by the duration of the subsequent vowel [53, 54]), requiring information to flow ‘right-to-left’. However, these ‘post-diction’ effects [55] seem to have made little impact on the field’s shared understanding. This may be in part because the mechanisms underlying right-to-left effects are unclear, and superficial explanations of these phenomena can be constructed that conform to parts of the STCF (see Box 2).

More recent empirical studies reveal the inadequacy of even a weak form of the STCF. Importantly, this work has not been synthesized. Thus, we review several lines of research in speech perception and word recognition that converge to challenge every aspect of the STCF. While none of these findings are definitive individually, they collectively call into question the STCF, raise the need for new frameworks, and provide new benchmarks for evaluating models.

Acoustic-phonetic representations do not decay rapidly

A key assumption of the STCF is that once activated representations are integrated into higher-level units, they decay to make room for future input. For instance, voice onset time is a primary cue for stop-voicing contrasts (e.g., /b/ vs. /p/; [56]). However, a word like pebble contains multiple voice onset times. Thus, voice onset time must be rapidly encoded, used to access representations, and then decay in time for the system to process the next one (Figure 1A; the “categorize-and-discard” assumption [42, 43]).

Interactive activation models implement this assumption explicitly with decay. While few make exact claims about how fast (e.g., in milliseconds) acoustic-phonetic (or lexical representations) decay, most models assume rapid decay (a fraction of a second) to keep up with the rate of speech input. In fact, when models have insufficient decay, they exhibit performance reminiscent of listeners with language disorders [57].

Recent research questions this assumption: information appears to be retained throughout processing for quite some time (Figure 1B). For instance, eye-tracking and brain imaging studies suggest that listeners retain acoustic-phonetic information about the onset of words like barricade and parakeet (the degree to which the initial sound is /b/ or /p/), and this influences listeners’ recovery from lexical mis-parses (when the -cade or -keet arrives; [58–60]). Critically, this is not simply the retention of categorical information—listeners preserve fine-grained acoustic representations. Other work pushes this further, finding that acoustically ambiguous pronouns can be disambiguated by sentence context many words late, even after 35 syllables [61, 62].

Listeners may discover that retaining such information is a useful strategy in these tasks. However, recent work suggests otherwise. A study using EEG to decode the influence of fine-grained acoustic information during sentence processing indicated that acoustic-phonetic representations persist for at least a second after sentence-final words, even in the absence of further information [5]. Other research has shown similar effects, suggesting that earlier words (e.g., in a sentence) influence recognition of similar words heard later [39, 63], even when this is of no apparent benefit. That is, in a sentence like dogs catch dolls, there is no obvious function for dogs to affect recognition of dolls (it could have just as easily been balls). Thus, not only do acoustic-phonetic representations maintain activation–even when not needed—but they can actively shape the interpretation of subsequent input.

These results challenge a critical STCF principle: that once information is rapidly transduced from the signal, representations are reset to make way for the next input. This work suggests representations may not substantively decay at all—fine-grained acoustic-phonetic information is retained across words to influence future interpretations, even when it is not relevant [42, 43]. In some cases, this retention could be functional—it could facilitate revision if a mis-parse is made, but current models of word recognition do not capture this. While many could (e.g., by changing decay parameters), this might destabilize other properties (e.g., interference from consecutive inputs, or an inability to segment speech). More importantly, these models would have abandoned the assumption that acoustic-phonetic representations decay rapidly, requiring a change in theory.

Acoustic-phonetic processing is not always immediate or incremental

In speech perception, cues to the same phoneme can be asynchronous—consonant voicing is cued by both voice onset time and the length of the subsequent vowel. The STCF can explain how listeners accommodate this: as soon as one cue (e.g., voice onset time₎ arrives, it partially activates lexical or sub-lexical representations. Later cues (e.g., vowel length) update these representations as the system converges on a decision (Figure 2A). This preserves the STCF in the face of right-context effects (Box 2). A wealth of studies [64–67] support this extension of immediacy and incrementality to speech (Box 1).

However, recent research demonstrates that incremental processing is not ubiquitous. That is, in some cases, initial cues are stored in a memory buffer, encapsulated from lexical processing until more information arrives. For example, in fricatives (/s/ vs. /ʃ/), listeners do not immediately use the initial spectral information to identify the sound. Instead, they wait until the subsequent vowel (Figure 2B; Box 1; [68]). In fact, during the fricative itself, listeners show no evidence of even knowing they heard a fricative at all (instead of a stop consonant) (though see [69]). This delay is remarkable given that the frication contains more reliable information about the identity of the fricative than the vowel portion [70, 71].

Further research has extended this result. Fricatives contain coarticulatory information that predicts the upcoming vowel, and listeners can use this information to predict the vowel’s identity [72–74]. Research capitalized on this by showing that listeners could immediately predict the upcoming vowel from the fricative, even as they replicated the buffering effect for identifying the fricative [75]. That is, listeners could (partially) identify a vowel from the frication, but not the fricative itself—their decisions came in the opposite order of the relevant segment’s arrival (c.f. [69]).

It is tempting to dismiss these findings as niche; indeed, they are restricted to a subset of English fricatives. However, the fact that any acoustic cues are not processed immediately and incrementally seriously challenges the STCF. There are no models of word recognition that produce buffered integration—a specialized architecture may be required to model it. Hybrids may also be possible, in which lexical processing is not incremental, but information in the buffer remains temporally organized. Moreover, research has started to extend these findings. Korean stop contrasts seem to be integrated less continuously than English stops [76], and there are consistent individual differences across listeners in how continuously they integrate information [77]. Acoustic-phonetic processing may simply be less incremental than long assumed.

Lexical representations may not be strictly temporally ordered

Work has also challenged the assumption that lexical representations reflect the temporal order of their constituents. Models often posit that peel has a /p/ in the first ‘slot’, /i/ in the second, and /l/ in the third. If peel were represented as just the presence of /p/, /i/, and /l/, then the lexical entry for peel and leap would be identical, and listeners could not differentiate them from each other (or from the nonword eelp; Figure 3). Most models of word recognition determine candidate activation by a match between internal temporal structure and the order of the input [10, 13, 15, 17].

However, mounting research suggests lexical entries do not have a strict order. Inspired by work on word reading [78, for a review], listeners consider anadromes as candidates: peel partially activates leap (Figure 3 [79]). That is, listeners use the onset /p/ and offset /l/ in peel as partial evidence for leap, suggesting that lexical representations do not code strict temporal order. This has been replicated with the Visual World Paradigm [80, 81] and priming [82–84] and extended across phonological contrasts [81, 85] and to multisyllabic words [83].

Importantly, lexical candidates that share phonemes but not vowel position (e.g., leaf and flea) are not preferentially activated [81]. This suggests representations are not simply a ‘bag of sounds’—having /f/, /l/, and /i/ as constituents does not make two words identical. This raises the possibility that a loose temporal structure remains, perhaps a coarse syllable frame or envelope, anchored by the vowel [75, 86, 87]. This may be further supported by encoding fine-grained acoustic differences (e.g., the /l/ in peel differs from that in leap). Though this retains some degree of order, this is a far cry from the temporal structure assumed by most STCF-inspired models.

Lexical candidates do not always decay, and access is not always immediate

The STCF was built on work examining lexical processing in ideal conditions. However, mounting research on challenging listening demonstrates that lexical access is not always immediate, and competitors may not fully decay. This work has used the Visual World Paradigm to examine word recognition when speech is heard in noise [88, 89], is spectrally degraded [90, 91], or is too quiet [92, 93]. This is complemented by research in populations with hearing loss [28, 90, 91, 94, 95]. Together, this has uncovered lawful variation in lexical competition and questions whether the STCF is a useful description for all listeners or all conditions.

For example, when Normal Hearing listeners hear speech in moderately challenging conditions, and in some Cochlear Implant users, lexical candidates do not fully decay. Instead, they remain active through the end of processing, even after the entire word is heard (Figure 4C–D). This pattern has been termed sustained activation [88, 91, 94]. Elderly listeners—even with typical hearing—also show evidence of this [96], and priming studies show the same in young adult normal hearing listeners exposed to diverse dialects [97].

A more severe challenge to the STCF arises in more difficult listening. Pre-lingually deaf cochlear implant users [90], children with moderate hearing loss who use hearing aids [95], and normal hearing listeners hearing highly spectrally degraded [90] or very quiet speech [92, 93] all delay the onset of lexical access entirely by 150–250 milliseconds, almost to the end of the word (Figure 4E–F). As a result, they show less onset competition, as by the time lexical access begins, they’ve heard sufficient information to rule out these alternatives. In this wait-and-see profile, the process of mapping sound to lexical items becomes uncoupled from the input—incoming speech does not activate representations immediately.

The contexts that elicit wait-and-see and sustained activation are frequent. Background noise, quiet speech, and spectral degradation are ubiquitous, and most people undergo aging with or without hearing loss. Thus, in many contexts and for many people, word recognition may take a form that is not consistent with the STCF [98].

Recent research has gone further. This work examined a large population of cochlear implant users, finding that these profiles represent underlying dimensions of word recognition. Individuals differ primarily in the degree to which they wait-and-see or sustain activation [28] and often adopt unique combinations of both. These same dimensions also characterize typical listeners across ages in both optimal listening conditions [28] and various types of challenges [99]. This argues that these dimensions are universal properties of the system.

The consequences of this for the STCF are severe. Not only does it imply that an individual listener might both delay recognition and maintain competition (questioning immediacy and decay), but it also suggests that every listener differs in these properties. That is, the STCF represents only a small region of a multi-dimensional space defined by differences in immediacy and decay.

Current models of spoken word recognition can handle sustained activation by tweaking decay processes [57], though in doing so, they may no longer be consistent with the STCF. It is unknown whether any model can handle wait-and-see, as delaying access entirely (even as speech input arrives) might not be possible without specialized architecture(s). Most importantly, however, no current model characterizes listeners, either individually or as a group, as an interaction between the degree to which they maintain candidates and delay recognition. To do so would abandon two core assumptions of the STCF: that information immediately activates lexical representations and that representations decay.

Beyond isolated words: pushing the STCF to the limit

Our review has focused on the timescales of processing single phonemes and words. However, when we consider larger timescales, the limits of the STCF (or even a weakened version) quickly become apparent. For example, work on prosodic influences on word recognition suggests that the rhythm of an entire sentence can bias competition [100] and alter how a sentence is segmented, causing words to appear or disappear [101, 102]. These prosodic expectations can be built over several minutes of discourse [103]. Moreover, some research indicates that listeners with hearing impairment occasionally discover they have mis-parsed a word or phrase and must rebuild their interpretation of an entire sentence, and do so using auditory memory [104]. These findings suggest that substantial information must be preserved; there must be ways to reactivate words from auditory memory, and the temporal span of information for recognizing a single word may be large. In the limit, it is not clear that these functions can be accommodated by even a weakened STCF.

Towards a new paradigm

The lines of work reviewed here collectively reveal the limits of the STCF: all three of its assumptions are often violated in word recognition and speech perception. While it is clear that immediacy and some notion of order must be built into the system for many situations, it is also clear that we need a richer paradigm. As we move towards this paradigm, it is important to ask what types of theories can handle temporally decoupled processing.

It is unclear whether variants of existing models can accommodate these effects. For instance, TRACE [15] can be adjusted such that activation is built slowly and phonetic representations do not decay rapidly. In principle, such changes might model wait-and-see, sustained activation, or the long-term retention of acoustic-phonetic detail. However, it is unclear if this would sacrifice other phenomena that TRACE successfully accounts for. Similarly, the loosely-ordered phoneme representations of TISK [11] might account for anadrome effects [c.f., 84] (though this has not been tested). Yet, this may break down from interference created by slowly decaying phonetic representations (see Outstanding Questions). Moreover, current models are unable to replicate buffering effects or explain why buffering is so apparent in fricatives.

More importantly, even if a parametric variant of a model was sufficient to account for things like wait-and-see, buffered processing, or loose temporal order, it would be built from a different set of guiding principles in which processing is not strongly coupled to the temporality of its input. It would reflect a different underlying theory, and the explanations of model behavior would reflect different assumptions. The STCF would no longer be a useful framework in this context.

What comes next, then? As a first step, we must revisit the criteria for evaluating models. If current models are a guide, it is often enough to demonstrate incremental processing and competition among candidates, occasionally with the addition of neighborhood density effects [11, 12, 17, 19]. The recently developed EARSHOT is a good example [12]. Its realistic acoustic-phonetic input and distributed recurrent architecture offers a potentially ideal test bed for the phenomena described here. However, its validation largely rested on its ability to demonstrate incremental competition—standard STCF phenomena. This is not a criticism of EARSHOT. Rather, it illustrates the need for richer model evaluation.

In addition to simulating standard STCF phenomena, we suggest models should be able to demonstrate anadrome activation [79, 81] as well as wait-and-see and sustained activation [28] (preferably as individual difference dimensions). Similarly, models should be able to gracefully handle slowly decaying representations without interference, and ideally simulate buffering of acoustic-phonetic information.

A new generation of models is likely needed. Here, we propose that it might be fruitful to privilege vowels, as both buffering and anadrome effects seem to be structured around them: listeners do not recognize fricatives until the vowel arrives, and anadromes with the same vowel location are preferentially activated relative to words with a vowel mismatch. Vowels might be moments in the signal that are particularly perceptually stable [86, 87, 105], or carry important indexical or contextual information (perhaps used for talker normalization [106]). Similar proposals have been made for the syllable [101]. This does not necessarily imply a need for specialized architectures—vowel-centric representations can be discovered as a natural solution by models when provided context-laden input [107].

Ultimately, it may be most useful to think of the STCF as an anchor from which to expand. As work on individual differences attests [28, 99], there may be regions of parameter space where the STCF holds as a good description of processing (e.g., normal hearing typical adults in quiet), but many for which it breaks down (e.g., listeners with hearing impairments). By conceptualizing word recognition dimensionally, we may discover principles that can delineate when it is useful to decouple from time in some way (e.g., sustained activation or wait-and-see) and when to adopt STCF-like processing. However, in this framework, the STCF describes only a subregion of the model space—it is not useful as an organizing paradigm.

Implications beyond speech and word recognition

Moving beyond the STCF may help integrate speech perception and word recognition with other domains of psycholinguistics. In written word recognition, for instance, spatial coding of letter position (analogous to time in spoken words) does not play an absolute role, as shown by transposed letter effects [78], suggesting a less ordered representation. In sentence processing, “transposed word” effects have also been observed [108], and good-enough [109] and schema-based [110] models downplay strict temporal ordering in favor of building a general sense of meaning [see 2, for an analogy in lexical organization]. Other work suggests that temporal encoding of non-speech auditory information is dependent on the suprasegmental properties of the languages of the listener (e.g., tonality [111]). That is, the temporal demands of one’s language may shape how strongly the overall auditory system develops to become coupled to the input. New paradigms thus offer the opportunity for unifying theories of temporal processing across levels of comprehension and modalities.

Concluding remarks

The work reviewed here suggests enormous diversity in how speech processing handles time: acoustic-phonetic cues may be integrated incrementally or in a buffer, information can be retained to different degrees across levels, lexical representations have no strict temporal order, and listeners vary in the degree to which lexical competitors are activated immediately and decay. The STCF represents a very small part of this space, and reliance on it as a unifying metaphor has impeded progress in understanding the mechanistic variety supporting recognition. Moreover, it has stymied understanding of variation across people. As work in cognitive science becomes increasingly sensitive to the effects of diversity in culture, language exposure, and experience on language acquisition and processing [22, 112], it is imperative to generate theories and models that capture this range. By moving on from the STCF, the next generation of models may better embrace the diversity of processes within speech and word recognition, and go beyond the modal listener, applying to more people under more circumstances [98].