Group Conversation Enhancement

This post accompanies two presentations titled “Immersive Conversation Enhancement Using Binaural Hearing Aids and External Microphone Arrays” and “Group Conversation Enhancement Using Wireless Microphones and the Tympan Open-Source Hearing Platform”, which were presented at the International Hearing Aid Research Conference (IHCON) in August 2022. The latter is part of a special session on open-source hearing tools.

Have you ever struggled to hear the people across from you in a crowded restaurant? Group conversations in noisy environments are among the most frustrating hearing challenges, especially for people with hearing loss, but conventional hearing devices don’t do much to help. They make everything louder, including the background noise. Our research group is developing new methods to make it easier to hear in loud noise. In this project, we focus on group conversations, where there are several users who all want to hear each other.

A group conversation enhancement system should turn up the voices of users in the group while tuning out background noise, including speech from other people nearby. To do that, it needs to separate the speech of group members from that of non-members. It should handle multiple talkers at once, in case people interrupt or talk over each other. To help listeners keep track of fast-paced conversations, it should sound as immersive as possible. Specifically, it should have imperceptible delay and it should preserve spatial cues so that listeners can tell what sound is coming from what direction. And it has to do all that while all the users are constantly moving, such as turning to look at each other while talking.

A normal hearing device on its own can’t distinguish speech from group members and non-members, much less enhance it while keeping up with rapid motion. To do that, we need help from extra devices that can isolate the talkers we want to hear. We will explore three strategies for tuning out background noise: using dedicated microphones clipped to each talker; using a large microphone array that can capture sound from a distance; and using mobile and wearable devices that users already have with them.

Once we have those low-noise signals from the talkers we care about, we need to process them to sound natural to the listener. Ideally, the output of the conversation enhancement should sound like what the earpiece microphones would have captured if there was no one else in the room. Last year, we proposed an adaptive signal processing method that matches the spatial cues of the low-noise signals to the sound at the earpieces. We will use that same method again here.

Remote Microphones

Wireless microphones are already widely used in classroom settings to transmit sound directly from the teacher to a student’s hearing device. Similar products are available for adults, but they aren’t very popular, in part because they can only be used with one talker at a time. Last year, we demonstrated a system for combining two or more wireless microphones for group conversations. It works even when multiple people talk over each other, and it preserves spatial cues to help listeners track the conversation. We implemented a two-talker version of the system in real time using the Tympan open-source hearing platform. The video below, originally presented at the Acoustical Society of America Meeting, demonstrates how the algorithm reduces noise while preserving spatial cues.

Microphone Arrays

It is often impractical to ask conversation partners to wear wireless microphones. Instead, we can use a microphone array to capture sound from a distance; that process is known as beamforming. Microphone arrays are commonly used in consumer devices such as smart speakers and, increasingly, in room infrastructure to support remote work and teaching. For example, our building’s main conference room has three microphone-array-equipped ceiling tiles. If these could be connected to hearing devices in the room, they could help people hear better during crowded events.

As part of our second talk at IHCON this week, we demonstrated a conversation enhancement system using a microphone array placed in the middle of a dining table. A group of three users wearing earpieces had a conversation while moving naturally, for example turning to look at each other. A set of three pre-calibrated beams isolate the talkers’ voices and send the enhanced data to the listening devices.

Cooperative Processing

Wireless microphones and microphone arrays both require users to deal with extra devices, which is likely part of the reason existing products are not more widely used. But in a typical group conversation, there are already several microphones spread among the group members. We could enhance the conversation without any extra gadgets by connecting the devices that users already have, including hearing devices, smartphones, and other wearables. They can share data with each other to achieve much better noise reduction than any device could alone.

We can demonstrate a simple version of cooperative processing by directly connecting two Tympan devices. The Tympan has a built-in microphone, so it can be used as a remote microphone with no extra hardware required. We can connect the line out of one Tympan to the line in of the other. The video below shows how the Tympan can be used as an immersive remote microphone. Only one Tympan was available when this demo was made, so it only shows one side of the conversation.

With fast wireless connections, we could also use smartphones as part of a cooperative processing network. Next month at the International Conference on Acoustic Signal Enhancement (IWAENC), we will present a method for combining earpieces with mobile devices placed on a table. It uses more complex adaptive algorithms to reduce noise and track motion. We demonstrate it for a group of three moving human talkers, just like in the array demonstration above.

Scaling up conversation enhancement

Ideally, group conversation enhancement would work with any number of users. The Tympan only has four analog input and output channels, so to scale the conversation enhancement system for larger groups, we will need to develop other means for the devices to share signals, such as by digital communication.

For closely spaced group conversations where all the microphones pick up sound from all the talkers, we also need a reliable way to tell which user is speaking when. That process is called voice activity detection (VAD). Normal earpiece microphones work surprisingly poorly for VAD because they don’t capture very much of their wearer’s voice at high frequencies. Lapel microphones and tabletop microphones are slightly better, but only when the talkers are spread far apart. Fortunately, hearing devices and high-tech earbuds include many other sensors, such as contact microphones that pick up vibrations through the body. These microphones have poor sound quality, but because they don’t capture sound from the air, they are very good at rejecting external noise.

The Tympan earpieces do not have any contact microphones built in, so we used an off-the-shelf throat microphone instead. The video below compares the sound from the contact microphone to that from the Tympan’s built-in microphone. The contact microphone sounds terrible in quiet, but it’s mostly unaffected by noise. It can therefore be used to detect who is talking when and update the adaptive filters accordingly.

Immersive Remote Microphone System on the Tympan Platform

This post accompanies our presentation “Immersive multitalker remote microphone system” at the 181st Acoustical Society of America Meeting in Seattle.

In our previous post, which accompanied a paper at WASPAA 2021, we proposed an improved wireless microphone system for hearing aids and other listening devices. Unlike conventional remote microphones, the proposed system works with multiple talkers at once, and it uses earpiece microphones to preserve the spatial cues that humans use to localize and separate sound. In that paper, we successfully demonstrated the adaptive filtering system in an offline laboratory experiment.

To see if it would work in a real-time, real-world listening system, we participated in an Acoustical Society of America hackathon using the open-source Tympan platform. The Tympan is an Arduino-based hearing aid development kit. It comes with high-quality audio hardware, a built-in rechargeable battery, a user-friendly Android app, a memory card for recording, and a comprehensive modular software library. Using the Tympan, we were able to quickly demonstrate our adaptive binaural filtering system in real hardware.

The Tympan processor connects to a stereo wireless microphone system and binaural earbuds.

Continue reading

Improving remote microphones for group conversations

This post accompanies the paper “Adaptive Binaural Filtering for a Multiple-Talker Listening System Using Remote and On-Ear Microphones” presented at WASPAA 2021 (PDF).

Wireless assistive listening technology

Hearing aids and other listening devices can help people to hear better by amplifying quiet sounds. But amplification alone is not enough in loud environments like restaurants, where the sound from a conversation partner is buried in background noise, or when the talker is far away, like in a large classroom or a theater. To make sound easier to understand, we need to bring the sound source closer to the listener. While we often cannot physically move the talker, we can do the next best thing by placing a microphone on them.

A remote microphone transmits sound from a talker to the listener's hearing device.

Remote microphones make it easier to hear by transmitting sound directly from a talker to a listener. Conventional remote microphones only work with one talker at a time.

When a remote microphone is placed on or close to a talker, it captures speech with lower noise than the microphones built into hearing aid earpieces. The sound also has less reverberation since it does not bounce around the room before reaching the listener. In clinical studies, remote microphones have been shown to consistently improve speech understanding in noisy environments. In our interviews of hearing technology users, we found that people who use remote microphones love them – but with the exception of K-12 schools, where remote microphones are often legally required accommodations, very few people bother to use them.

Continue reading

Dynamic Range Compression and Noise

This post accompanies our presentation “Dynamic Range Compression of Sound Mixtures” at the 2020 Acoustical Society of America meeting and our paper “Modeling the effects of dynamic range compression on signals in noise” in the Journal of the Acoustical Society of America (PDF).

Nearly every modern hearing aid uses an algorithm called dynamic range compression (DRC), which automatically adjusts the amplification of the hearing aid to make quiet sounds louder and loud sounds quieter. Although compression is one of the most important features of hearing aids, it might also be one of the reasons that they work so poorly in noisy environments. Hearing researchers have long known that when DRC is applied to multiple sounds at once, it can cause distortion and make background noise worse. Our research team is applying signal processing theory to understand why compression works poorly in noise and exploring new strategies for controlling loudness in noisy environments.

Continue reading

Face masks make it harder to hear, but amplification can help

This post describes our recent paper “Acoustic effects of medical, cloth, and transparent face masks on speech signals” in The Journal of the Acoustical Society of America. This work was also discussed in The Hearing Journal and presented at the 179th Acoustical Society of America meeting.

Face masks are a critical tool in slowing the spread of COVID-19, but they also make communication more difficult, especially for people with hearing loss. Face masks muffle high-frequency speech sounds and block visual cues. Masks may be especially frustrating for teachers, who will be expected to wear them while teaching in person this fall. Fortunately, not all masks affect sound in the same way, and some masks are better than others. Our research team measured several face masks in the Illinois Augmented Listening Laboratory to find out which are the best for sound transmission, and to see whether amplification technology can help.

Several months into the pandemic, we now have access to a wide variety of face masks, including disposable medical masks and washable cloth masks in different shapes and fabrics. A recent trend is cloth masks with clear windows, which let listeners see the talker’s lips and facial expressions. In this study, we tested a surgical mask, N95 and KN95 respirators, several types of opaque cloth mask, two cloth masks with clear windows, and a plastic face shield.

Face masks ranked by high-frequency attenuation

We measured the masks in two ways. First, we used a head-shaped loudspeaker designed by recent Industrial Design graduate Uriah Jones to measure sound transmission through the masks. A microphone was placed at head height about six feet away, to simulate a “social-distancing” listener. The loudspeaker was rotated on a turntable to measure the directional effects of the mask. Second, we recorded a human talker wearing each mask, which provides more realistic but less consistent data. The human talker wore extra microphones on his lapel, cheek, forehead, and in front of his mouth to test the effects of masks on sound capture systems.

Continue reading

What do you want from hearing tech?

Our team at the Illinois Augmented Listening Laboratory is developing technologies that we hope will change the way that people hear. But the technology is only one half of the story. If we want our research to make a difference in people’s lives, we have to talk to the people who will use that technology.

Our research group is participating in the National Science Foundation Innovation Corps, a technology translation program designed to get researchers out of the laboratory to talk to real people. By understanding the needs of the people who will benefit from our research, we can make sure we’re studying the right problems and developing technology that will actually be used. We want to hear from:

  • People with hearing loss who use hearing aids and assistive listening devices
  • People who don’t use hearing technology but sometimes have trouble hearing
  • Parents, teachers, school administrators, and others who work with students with hearing loss
  • Hearing health professionals
  • People who work in the hearing technology industry

This is not a research study: there are no surveys, tests, or consent forms. We want to have a brief, open-ended conversation about your needs, the technology that you use now, and what you want from future hearing technology.

To schedule a call with our team, please reach out to Ryan Corey (corey1@illinois.edu). Most calls last about 15 minutes and take place over video, though we’re happy to work around your communication needs.

Hearing aid algorithm adapted for COVID-19 ventilators

Audio signal processing would seem to have nothing to do with the COVID-19 pandemic. It turns out, however, that a low-complexity signal processing algorithm used in hearing aids can also be used to monitor breathing for patients on certain types of ventilator.

To address the shortage of emergency ventilators caused by the pandemic, this spring the Grainger College of Engineering launched the Illinois RapidVent project to design an emergency ventilator that could be rapidly and inexpensively produced. In little more than a week, the team built a functional pressure-cycled pneumatic ventilator, which is now being manufactured by Belkin.

The Illinois RapidVent is powered by pressurized gas and has no electronic components, making it easy to produce and to use. However, it lacks many of the monitoring features found in advanced commercial ventilators. Without an alarm to indicate malfunctions, clinicians must constantly watch patients to make sure that they are still breathing. More-advanced ventilators also display information about pressure, respiratory rate, and air volume that can inform care decisions.

The Illinois RapidAlarm adds monitoring features to pressure-cycled ventilators.

To complement the ventilator, a team of electrical engineers worked with medical experts to design a sensor and alarm system known as the Illinois RapidAlarm. The device attaches to a pressure-cycled ventilator, such as the Illinois RapidVent, and monitors the breathing cycle. The device includes a pressure sensor, a microcontroller, a buzzer, three buttons, and a display. It shows clinically useful metrics and sounds an audible alarm when the ventilator stops working. The hardware design, firmware code, and documentation are available online with open-source licenses. A paper describing how the system works is available on arXiv.

Continue reading

Audio Source Remixing

This post describes our paper “Binaural Audio Source Remixing for Microphone Array Listening Devices” presented at ICASSP 2020. You can read the paper here or watch the video presentation here.

Microphone arrays are important tools for spatial sound processing. Traditionally, most methods for spatial sound capture can be classified as either beamforming, which tries to isolate a sound coming from a single direction, or source separation, which tries to split a recording of several sounds into its component parts. Beamforming and source separation are useful in crowded, noisy environments with many sound sources, and are widely used in speech recognition systems and teleconferencing systems.

Since microphone arrays are so useful in noisy environments, we would expect them to work well in hearing aids and other augmented listening applications. Researchers have been building microphone-array hearing aids for more than 30 years, and laboratory experiments have consistently shown that they can reduce noise and improve intelligibility, but there has never been a commercially successful listening device with a powerful microphone array. Why not?

The problem may be that most researchers have approached listening devices as if they were speech recognition or teleconferencing systems, designing beamformers that try to isolate a single sound and remove all the others. They promise to let the listener hear the person across from them in a crowded restaurant and silence everyone else. But unlike computers, humans are used to hearing multiple sounds at once, and our brains can do a good job separating sound sources on their own. Imagine seeing everyone’s lips move but not hearing any sound! If a listening device tries to focus on only one sound, it can seem unnatural to the listener and introduce distortion that makes it harder, not easier, to hear.

A source remixing system changes the relative levels of sounds in a mixture while preserving their spatial cues.

This paper proposes a new type of array processing for listening devices: source remixing. Instead of trying to isolate or separate sound sources, the system tries to change their relative levels in a way that sounds natural to the listener. In a good remixing system, it will seem as if real-world sounds are louder or quieter than before.

Continue reading