A cochlear implant (CI) is a neuro-prothesis that has helped over 1 million deaf people worldwide restore their hearing. This is achieved by transmitting sound vibrations into electrical stimulation that (re)activates cochlear cells to send neural impulses to the auditory cortex. However, electrical stimulation is artificial which does not convey sufficient auditory inputs as in typical hearing. Speech communication remains a huge challenge for CI recipients in day-to-day complex, noisy listening environments (at school/work, in streets/restaurants/train stations, meeting with families and friends). CI recipients often use visual speech cues (e.g., lip-reading) to compensate for these auditory barriers to help with communications. However, mechanisms underlying how their different sensory (auditory and visual) systems work together to integrate audiovisual speech are still unclear. It also remains unclear how interactions between these different systems change or 'reorganise' over time and how reorganisation predicts future speech comprehension outcomes.

The project promises to conduct a series of cross-sectional and longitudinal experiments in adult CI recipients. We will combine brain imaging of electroencephalography (EEG) (real-time neural activity with millisecond precision) and high-density fNIRS/diffuse optical tomography (HD-DOT) (recording blood oxygenation level at specific brain regions) to measure CI recipients’ neural responses to audio and visual speech – how they may differ from typical hearing, change over time and be modulated to improve comprehension overcomes. We will thus, scientifically, showcase example processes of cross-modal neuroplasticity (how sensory systems with different modalities, e.g., audition and vision, change the way they interact with each other in the brain) in humans with sensory impairment or deprivation after rehabilitation.

Wider implications, e.g., outcomes from multimodal imaging and clinical practicality (not just for CI recipients) may be expected. We may provide evidence for the potential power for monitoring and prognosis of speech and language comprehension outcomes using tools like EEG combined with portable HD-DOT compared to techniques commonly used but less physically or financially practical/accessible – e.g., MRI/fMRI, under many circumstances – low-noise requirement for auditory screening, physical/mental challenges for children/patients (e.g., staying still and focused for a long period in an enclosed environment), MRI's incompatibility with implanted electronic/metallic devices, public fiscal affordability of large-scale generic/longitudinal (f)MRI screening, requirement of portability, bedside monitoring, etc. This is particularly useful given the insufficient prognostic power relying solely on behavioural measures and for many whose behavioural data are not easy to be reliably captured to reflect actual speech comprehension. We may also provide rigorous, consequential neuromodulation evidence for future techniques researchers could develop to support existing speech and language therapies; in the longer run, for not only people with hard-of-hearing but also those with conditions like aphasia, developmental disorders, neurodivergence, or neurodegeneration.