Excess Vowel Fusion in Hearing-Impaired Listeners Across Fundamental Frequency and Spatial Separation in the Free Field: An Alternative Mechanism for Difficulties With Speech in Noise

Participants

These studies were conducted according to the guidelines for the protection of human subjects as set forth by the Institutional Review Board (IRB) of Oregon Health and Science University (OHSU), and the methods employed were approved by that IRB. Eleven NH and eleven HI listeners were recruited for the study. One NH listener was excluded based on a poor pure-tone average and one HI listener was excluded based on poor monaural vowel identification. Ultimately, ten NH listeners ranging in age from 21 to 43 years (mean 27.0 ± 7.5 years; 9 females) and ten bilateral HA users ranging in age from 21 to 71 years (mean 40.8 ± 18.8 years; 4 females) participated in this study. There was no significant difference in age between groups (p>0.05, two-tailed t-test). All subjects were screened for normal cognitive function using the Mini Mental Status Examination with a minimum score of 27 out of 30 required to qualify (MMSE; Folstein et al., 1975; Souza et al., 2007).

NH subjects were screened for audiometric thresholds within normal limits (thresholds ≤ 20 dB HL) from 250 to 4000 Hz. Audiometric thresholds were screened up to 4000 Hz because all testing was conducted with tones and relevant vowel F1 and F2 information below 4000 Hz. Group average audiograms for NH listeners and individual audiograms for HI listeners are shown for left and right ears in Figure 1. Demographics for HI listeners, along with duration of hearing loss, duration of HA use, and daily hours of HA use, are shown in Table 1.

All subjects were native speakers of American English. NH listeners were required to meet criterion performance of 84% correct or better on monaural single vowel perception for both ears (procedure described below). Vowel perception score criteria were less stringent for HI listeners, with a larger range between 49.0–87.5 percent correct. The majority of the NH subjects grew up in the Pacific Northwest or California; the two exceptions were a participant from the South (Texas) and a participant from the Northeast (Pennsylvania). The HI subjects similarly came mainly from the Pacific Northwest (Table 2), with the exception of HI45 who was from Mexico, but English was their first language.

Stimuli and Procedures

All experiments were conducted in a double-walled, sound-attenuated booth. Signals were generated at a sampling rate of 44.1 kHz and processed through an external soundcard (RME Babyface). For binaural tone fusion measurements, stimuli were generated using Matlab (version R2010b) and presented using Sennheiser HD-25 headphones. For vowel perception experiments, stimuli were generated by a Klatt speech synthesizer implemented in Praat (Boersma & Weenick, 2016) and presented through two active loudspeakers (Avantone MixCube) placed symmetrically 38 inches away at ±60° relative to a seated listener’s head (Figure 2A).

Vowel Fusion and Identification

Results were quantified using two metrics. First, the percentage of vowel fusions was measured as the percent of single vowel responses when different vowels were presented, as a function of fundamental frequency difference. Second, overall vowel identification performance was assessed as the percent of double vowel trials on which both vowels were correctly identified.

Figure 4 shows the percentage of two-vowel trials on which listeners gave a single-vowel response as a function of ΔF₀ in semitones for NH (left panel) and HI (right panel) listeners. Listeners in both groups tended to give single-vowel responses more frequently when F₀ was the same for both vowels in a trial; as the ΔF₀ increased, many listeners—especially listeners with normal hearing—identified two vowels on each trial.

Significant main effects were observed for ΔF0 (F (1.07, 119.17) = 87.29, p < 0.001), and listener group (F (1, 18) = 6.88, p = 0.017), but no significant interaction was observed between ΔF₀ and listener group (F (1.07, 19.17) = 1.74, p = 0.20). Post hoc tests showed significant differences between ΔF₀ = 0 and ΔF₀ = 6 (p<0.001) but not between ΔF₀ = 6 and ΔF₀ = 11 (p = 0.36).

Examination of responses to the single-vowel catch trials indicated that most listeners, regardless of listener group, gave single-vowel responses on the majority (>90%) of trials where the same stimulus was presented from each speaker. The exception was listener HI57, who provided two-vowel responses on 61.25% of the single vowel trials. However, removing this participant from the statistical analyses did not change the pattern of significant effects observed.

Figure 5 shows the percent correct identification of both vowels presented on double-vowel trials as a function of ΔF₀ in semitones for NH (left panel) and HI (right panel) listeners. Correct identification of both vowels tended to be poor for both listener groups when ΔF₀ was the same. As ΔF₀ increased, identification performance increased for listeners with normal hearing, although an increase in ΔF0 from 6 to 11 semitones did not result in additional improvement. For listeners with hearing loss, increasing ΔF₀ improved identification accuracy to a lesser degree—in some cases, not at all.

Significant main effects were observed for ΔF₀ (F (1.09, 19.67) = 64.72, p < 0.001), and listener group (F (1, 18) = 27.82, p < 0.001); additionally, a significant interaction was observed between ΔF₀ and listener group (F (1.09, 19.67) = 4.72, p = 0.039). Post hoc tests showed significant differences between ΔF₀ = 0 and ΔF₀ = 6 overall (p<0.001) and between the listener groups (p = 0.033), but not between ΔF₀ =6 and ΔF₀ =11 either overall (p = 0.322) or between the groups (p = 0.465).

Correlation of Vowel Fusion With Tone Fusion

Figure 6 shows the relationship between the percentage of single-vowel responses and the tone fusion range for each listener in the current study. No correlation was observed between single-vowel responses and fusion range for NH listeners alone (R (10) = 0.03, p = 0.928), HI listeners alone (R (10) = 0.07, p = 0.853), or all listeners considered together (R (20) = 0.29, p = 0.222).

Confusion Matrices

Confusion matrices were created to compare the vowel pairs presented to listeners’ vowel response(s). Figures 7 and 8 show the group averages confusion matrices (CMs) for the trials where F₀ was the same for both vowels and for the trials where F₀ was different, respectively. The CMs on the left indicate the pooled responses for the NH listeners and the CMs on the right indicate the pooled responses for the HI listeners. Each column indicates a vowel pair presented, and each row indicates a possible response to that pair. The first eight rows indicate single-vowel responses—including the four target categories (bold) and the four non-target categories (italic). The next twenty-two rows indicate two-vowel responses where one or both vowel responses were non-target categories. The last six rows indicate two-vowel responses where both target vowels were correctly identified. The numbers and darkness of shading in each cell indicate the percentage of each response combination selected in response to a vowel pair.

As can be seen in Figure 7, when F₀ was the same (ΔF₀ = 0), the majority of responses made by both listener groups were single vowels and these single responses tended to be one of the two vowels presented—especially for the NH listeners. Additionally, the NH listeners were more likely than HI listeners to make two-vowel responses. This contrasts with the results of Reiss and Molis (2021) where both listeners groups made very few two-vowel responses when ΔF₀ = 0. When the single responses were not one of the two vowels presented, the intermediate vowel response options, i.e. those in between the presented vowels at the edges of the F1-F2 vowel space (Figure 3), tended to be selected, especially for the HI listeners. For instance, the vowel/ʌ/was often perceived for vowel pairs with /a/ (Figure 7, columns 1-3 of right panel for HI listeners), and the vowels/ɛ/and/I/were sometimes perceived for vowel pairs containing /æ/or /i/ (Figure 7, columns 4 and 6 of left panel for NH listeners and columns 3-6 of right panel for HI listeners).

In addition, the NH listeners showed some consistent errors for the target vowel /æ/. For the three vowel pairs containing the target category /æ/, seven of the ten NH listeners displayed an increased likelihood to identify /æ/ as the non-target category /ɛ/. This pattern was also observed in one of the HI listeners. This confusion was not observed in the diotic training or the monaural screening, in which NH listeners were able to reliably distinguish between /æ/ and /ɛ/.

When F₀ differed (ΔF₀>0) (Figure 8), the two patterns of vowel confusion varied somewhat between the two groups. Similar to the response patterns observed previously, both vowels of a pair were more often identified correctly by subjects in the NH group, while HI subjects were less likely to make two-vowel responses than NH listeners. Similar patterns of intermediate vowel perception were observed for the HI listeners when F₀ differed (ΔF₀>0) as when F₀ was the same (ΔF₀ = 0; Figure 8, column 1 and columns 3-6 of right panel for HI listeners), though present to a lesser extent.

Vowel Fusion and Identification in Hearing-Impaired and Normal-Hearing Listeners

The current study measured vowel fusion in the free field for NH and HI listeners. We wished to assess whether patterns of vowel fusion previously observed under dichotic headphone conditions by Reiss and Molis (2021) would generalize to more natural listening conditions. There were some similarities in the response patterns observed under headphones and in the free field: when ΔF₀ = 0, single-vowel responses were most common, and identification accuracy was lowest; as ΔF₀ increased, the number of single-vowel responses dropped, and identification accuracy increased.

In contrast to the results observed under headphones by Reiss and Molis, the NH listeners in the current study were less likely to give single-vowel responses when ΔF₀ = 0 (see Figure 4). Previously, over headphones, almost all NH listeners gave single-vowel responses on >80% of the trials when F₀ was the same, and as ΔF₀ increased, they gave fewer single-vowel responses. In the current study, under free-field conditions, when ΔF₀ = 0 the NH listeners made fewer single-vowel responses than were made in the previous study, and a few of the NH listeners gave single-vowel responses on fewer than half of the trials where F₀ was the same. Additionally, the number of single-vowel responses fell to a greater degree with increased ΔF₀ than in the previous study.

Similar to the results of the previous study under headphones, HI listeners were more likely than NH listeners to make single-vowel responses when ΔF₀ = 0. However, two of the HI listeners (HI43 and HI61) gave single-vowel response on only about 60% of the trials where ΔF₀ = 0; and although half the HI listeners continued to make frequent single-vowel responses as F₀ difference increased, for the other half of HI listeners, the number of single-vowel responses fell to about the same level as for the NH listeners as ΔF₀ increased.

Also, in contrast to the results of Reiss and Molis (2021), in the current study a few of the NH listeners were able to identify both vowels with fair accuracy (20–40% both vowels correct) when F₀ was the same (Figure 5). Surprisingly, however, the improvement in identification accuracy observed currently with increasing ΔF₀ appeared to be somewhat less than that observed previously. HI listeners performed poorly on vowel identification when the F₀ was the same and generally improved as ΔF₀ increased; however, similar to the NH listeners, they did not appear to experience the same degree of benefit from increases in ΔF₀ as had been previously observed.

In general, these findings are consistent with the improvement seen with the introduction of time difference cues as in Eddolls et al. (2022), where both NH and HI listeners decreased vowel fusion and improved accuracy in identification of both of the dichotic vowels when onset asynchrony of 1 msec or more was added. While no significant improvement in identification was seen with the interaural time differences present for two copies of each vowel (Shackleton et al., 1994; Shackleton & Meddis, 1992), those studies may have been underpowered to detect a significant effect, as a small improvement is observed in both studies. The lack of effect of level differences shown in previous studies (Fan et al., 2022; Shackleton et al., 1994; Shackleton & Meddis, 1992) suggests that the effects may be mainly due to the time differences.

The original study had four target vowel categories and four response categories. Although the original four target vowels (/a/, /æ/, /i/, and /u/) were not the best exemplars, the limited response categories appeared adequate, and participants labelled them as intended. However, concerns about the quality of the original target vowels led us to optimize the stimuli for quality with F1 and F2 values closer to the centroids in the F1-F2 vowel space for adult male and female speakers of American English (Hillenbrand et al., 1995). In addition, in the previous study participants were asked to choose the response that most closely matched the percept, but some noted they heard an option not provided due to the limited number of response options. In the current study, to assess if the fused percepts better corresponded to other vowel category labels, we expanded the number of response options to include vowel categories (/I/, /ɛ/, /ʌ/, and /ʊ/) that were intermediate between the original target categories, i.e. might reflect a perceptual averaging of the peaks of the constituent vowels. Because we added new potential response options, we also added corresponding category exemplars to the familiarization. Either or both of these changes may have led to greater reporting of these intermediate vowel percepts, especially in the HI listeners who had greater difficulty with the larger number of response options, as evidenced by the lower identification scores for single vowels compared to the original study. Alternatively, presentation in the free field resulting in two copies of the vowels with various level and time differences at each ear may have allowed for different manifestations of partial fusion and averaging than seen under headphone presentation. Further studies are needed with direct comparison between headphone and loudspeaker conditions to separate out changes due to listener, apparatus condition, and number of response options.

Additional non-target stimuli corresponding to intermediate vowels were also added to testing as diotic catch trials where the non-target vowels were presented simultaneously from both speakers. These catch trials were introduced to provide listeners with trials with legitimate single-vowel percepts to contrast with potential illusory fused-vowel percepts of the two-vowel trials, and to screen for listeners responding to ambiguous percepts by inferring the presence of two vowels and selecting two vowels in response, as shown previously when listeners are informed that there are two vowels presented (Kwon & Perry, 2014). Analysis of the responses to the catch trials for both listener groups in this sample indicated overwhelmingly single-vowel responses, ruling out this scenario.

We also speculated that the presence of the single-vowel trials could significantly disrupt the perception of fused vowels if listeners had unambiguous examples of true single-vowel presentations. This did not appear to be the case. Although the number of fused percepts in the two-vowel trials was reduced, they were not eliminated. This may indicate that overall poor or ambiguous exemplar quality in the previous study was not a reliable cue that two different vowels were presented.

Relationship of Dichotic Vowel Fusion to Dichotic Tone Fusion

Previously, Reiss and Molis (2021) observed a significant relationship between the percentage of single-vowel responses (fused vowel percept) and the tone fusion range (fused tone percept) for NH listeners, but not for HI listeners. When the relationship was considered across all listeners, the overall correlation of tone fusion range with percent single-vowel responses was significant. Similarly, Oh et al. (2022) observed a strong correlation was observed between harmonic tone fusion range and the benefit from voice pitch differences (voice gender release from masking) for connected speech in free-field environments in for both NH and HI listeners, in which tone fusion range explained as much as 76% of the variance.

In the current study, no significant correlations were observed between the percentage of single-vowel responses and tone fusion range. There are several possible explanations for this discrepancy from the previous two studies. First, there was a narrower spread of tone fusion ranges among the NH listeners and the HI listeners in the current study. Second, there were fewer single-vowel responses overall, leading to a smaller spread of single vowel response rates. Third, there was more scatter in both dimensions, suggesting that other factors present under free field listening conditions, such as interaural timing and level cues, may contribute. Finally, the Oh et al. (2022) study measured binaural fusion range using complex tones in the voice pitch range of 100–200 Hz rather than with pure tones at 2000 Hz as in the current study; fusion ranges of complex tones in the voice pitch range may be more closely linked to fusion of voices of different pitch in a real-world listening task as in Oh et al. (2022), and may explain the much stronger correlations seen in that study.

Vowel Confusion Patterns Differ for Trials Where ΔF₀ = 0 Versus ΔF₀ > 0

The basic patterns in vowel identification with differences in F0 that were observed in the previous study were also observed in the current study: mostly single-vowel responses were given when F₀ was the same and mostly two-vowel responses were given when F₀ differed, and the benefit of F₀ differences was greater for the NH listeners than for the HI listeners.

As noted above, the current study provided additional response options that may have influenced responses slightly for some listeners. Previously, in Reiss and Molis (2021), we observed fused responses that differed from either vowel presented, i.e., new vowels not in the original pair were heard—/a/ and /i/ fused into /ae/, or /ae/ and /u/ fused into /a/. Some of these fusions still occurred in the current study, but it appeared more common that the single-vowel response was one of the two vowels presented or a vowel category with a slightly higher or lower F1 than one of the target vowels, i.e. one of the intermediate vowel categories. As in the previous study, these fused and intermediate vowel percepts still occurred even when ΔF₀ differed across hemifields for HI listeners, indicating negative effects of vowel fusion on vowel perception and extending the applicability of the findings of Reiss and Molis (2021) to the free field and realistic multi-talker listening situations with spatial separation between talkers.

The inclusion of additional response categories in the current study also led to an interesting pattern of errors in the responses of many of the NH listeners, in which there was an increased likelihood to identify /æ/ when presented as part of a vowel pair as the non-target category /ɛ/. This tendency for /æ/→/ɛ/ in the dichotic trials cannot be explained by listeners’ single-vowel training results. One possibility is that this reflects fusion and averaging of two dichotically presented vowels as observed previously. Alternatively, context effects may occur due to the lack of actual /ɛ/ stimuli for contrast in the dichotic testing, shifting the perception of /æ/ to/ ɛ/. Another possibility is that dialect differences were unmasked and identification drifted during testing in the absence of feedback, as most of the participants originated from the west coast of the U.S. where the F1 and F2 formant values for/æ/as perceived in the midwest and east coast is heard as /ɛ/ in the vowel space (Wright & Souza, 2012). Future testing with the addition of /ɛ/ vowels to the dichotic testing and vowel formant values aligned with the west coast vowel space would be needed to rule out the latter two explanations.

Relationship of Findings to Perception of Connected Speech in the Free Field

As noted, binaural fusion can be considered as a subcategory of auditory grouping—specific to simultaneous and across-ear grouping. Hence, binaural fusion of tones and static vowels has relevance for this aspect of grouping of connected speech in free field listening environments. The confirmation of vowel fusion and spectral averaging in the free field in the current study supports the interpretation that the loss of benefit from voice pitch differences, i.e. reduced voice gender masking release, observed for connected speech in Oh et al. (2022) is due to excessive binaural fusion. This interpretation is supported by participants’ reported perception of non-words, such as “grain” instead of “red” or “green”, that appear to be result of fusion and spectral averaging.