An Unofficial Port of Matrix HAL to Ubuntu 22.04 and Raspbian Bullseye

What is Matrix VOICE and Matrix HAL?

The Matrix VOICE is described on the Matrix website as a “development board for building sound driven behaviors and interfaces.” It is a nifty piece of hardware that features an 8-microphone array and has unique possibilities for beamforming and audio processing applications.

However, in February 2021, when Matrix Labs was bought out, development and support for the Matrix VOICE C++ library, the Matrix HAL, was silently withdrawn.

The latest release of Matrix HAL only works with a version of the Raspberry Pi OS called Raspbian Buster which is fine for a lot of applications, but recently our research has been focused on integrating this device with ROS, a robotics framework that is meant for Ubuntu. While there were workarounds for using ROS on Raspbian Buster, this would not be without its own unique set of challenges.

It was determined that it would be valuable to investigate patching the Matrix HAL to work on Ubuntu 22.04. And over the summer, we were able to accomplish this goal.

Description of the Port

There are some limitations to the port. With our own acoustic research being the primary application, we did not ensure the functionality of sensors other than the microphone array. This means there is currently no support for the humidity sensor, IMU, pressure sensor or UV sensor. There is currently only support for the microphone array and the Everloop LED interface.

Additionally, the repo has not been tested on the Matrix CREATOR and there is no guarantee that it will be compatible with this patch.

You can download the source from this Github repository. Please direct any support related inquiries to gfw3@illinois.edu.

Enhancing Group Conversations with Smartphones and Hearing Devices

This post describes our paper “Adaptive Crosstalk Cancellation and Spatialization for Dynamic Group Conversation Enhancement Using Mobile and Wearable Devices,” presented at the International Workshop on Acoustic Signal Enhancement (IWAENC) in September 2022.

One of the most common complaints from people with hearing loss – and everyone else, really – is that it’s hard to hear in noisy places like restaurants. Group conversations are especially difficult since the listener needs to keep track of multiple people who sometimes interrupt or talk over each other. Conventional hearing aids and other listening devices don’t work well for noisy group conversations. Our team at the Illinois Augmented Listening Laboratory is developing systems to help people hear better in group conversations by connecting hearing devices with other nearby devices. Previously, we showed how wireless remote microphone systems can be improved to support group conversations and how a microphone array can enhance talkers in the group while removing outside noise. But both of those approaches rely on specialized hardware, which isn’t always practical. What if we could build a system using devices that users already have with them?

We can connect together hearing devices and smartphones to enhance speech from group members and remove unwanted background noise.

In this work, we enhance a group conversation by connecting together the hearing devices and mobile phones of everyone in the group. Each user wears a pair of earpieces – which could be hearing aids, “hearables”, or wireless earbuds – and places their mobile phone on the table in front of them. The earpieces and phones all transmit audio data to each other, and we use adaptive signal processing to generate an individualized sound mixture for each user. We want each user to be able to hear every other user in the group, but not background noise from other people talking nearby. We also want to remove echoes of the user’s own voice, which can be distracting. And as always, we want to preserve spatial cues that help users tell which direction sound is coming from. Those spatial cues are especially important for group conversations where multiple people might talk at once.

Continue reading

Turning the Music Down with Wireless Assistive Listening Systems

This post accompanies our presentation “Turn the music down! Repurposing assistive listening broadcast systems to remove nuisance sounds” from the Acoustical Society of America meeting in May 2022.

It is often difficult to hear over loud music in a bar or restaurant. What if we could remove the annoying music while hearing everything else? With the magic of adaptive signal processing, we can!

To do that, we’ll use a wireless assistive listening system (ALS). An ALS is usually used to enhance sound in theaters, places of worship, and other venues with sound systems. It transmits the sound coming over the speakers directly to the user’s hearing device, making it louder and cutting through noise and reverberation. Common types of ALS include infrared (IR) or frequency modulation (FM) transmitters, which work with dedicated headsets, and induction loops, which work with telecoils built into hearing devices.

We can instead use those same systems to cancel the sound at the ears while preserving everything else. We use an adaptive filter to predict the music as heard at the listener’s ears, then subtract it out. What’s left over is all the other sound in the room, including the correct spatial cues. The challenge is adapting as the listener moves.

The video below demonstrates the system using a high-end FM wireless system. The dummy head wears a set of microphones that simulate a hearing device; you’ll be hearing through its ears. The FM system broadcasts the same sound being played over the speakers. An adaptive filter cancels it so you can hear my voice but not the music.

Group Conversation Enhancement

This post accompanies two presentations titled “Immersive Conversation Enhancement Using Binaural Hearing Aids and External Microphone Arrays” and “Group Conversation Enhancement Using Wireless Microphones and the Tympan Open-Source Hearing Platform”, which were presented at the International Hearing Aid Research Conference (IHCON) in August 2022. The latter is part of a special session on open-source hearing tools.

Have you ever struggled to hear the people across from you in a crowded restaurant? Group conversations in noisy environments are among the most frustrating hearing challenges, especially for people with hearing loss, but conventional hearing devices don’t do much to help. They make everything louder, including the background noise. Our research group is developing new methods to make it easier to hear in loud noise. In this project, we focus on group conversations, where there are several users who all want to hear each other.

Conversation enhancement allows users within a group to hear each other while tuning out background noise.

A group conversation enhancement system should turn up the voices of users in the group while tuning out background noise, including speech from other people nearby. To do that, it needs to separate the speech of group members from that of non-members. It should handle multiple talkers at once, in case people interrupt or talk over each other. To help listeners keep track of fast-paced conversations, it should sound as immersive as possible. Specifically, it should have imperceptible delay and it should preserve spatial cues so that listeners can tell what sound is coming from what direction. And it has to do all that while all the users are constantly moving, such as turning to look at each other while talking.

Continue reading