Sound source localization and tracking using ManyEars library

Hello all,

I’ve just gotten ahold of a matrix creator kit and have been playing around with the microphone array. I’d like to use the ManyEars library [1] for Sound source localization and tracking to improve beamforming and engage with multiple users. Eventually I’ll add this to mobile robot for people tracking and group interaction using the ROS package [2], but for now I’m just trying to check the microphone array’s performance and suitable use with the library.

Problem

I’ve got malos all set up and can record audio from each microphone simply enough as described here:

However the structure of the alsa device seems strange as there are a number of dummy devices, and none of the microphones seem to be associated with input channels for any of the devices found.

$ arecord -L
null
    Discard all samples (playback) or generate zero samples (capture)
pulse
    PulseAudio Sound Server
mic_channel0
mic_channel1
mic_channel2
mic_channel3
mic_channel4
mic_channel5
mic_channel6
mic_channel7
mic_channel8
sysdefault:CARD=Dummy
    Dummy, Dummy PCM
    Default Audio Device
dmix:CARD=Dummy,DEV=0
    Dummy, Dummy PCM
    Direct sample mixing device
dsnoop:CARD=Dummy,DEV=0
    Dummy, Dummy PCM
    Direct sample snooping device
hw:CARD=Dummy,DEV=0
    Dummy, Dummy PCM
    Direct hardware device without any conversions
plughw:CARD=Dummy,DEV=0
    Dummy, Dummy PCM
    Hardware device with all software conversions

This causes some issues using library’s celebration/tunning tool that checks alsa device and channels to use:

I have a fork that includes the few CFLAGS changes I made to compile on ARM for the Raspberry Pie 2 [3], with the build only requiring a few packages, such as cmake libqt4-dev build-essential libasound2-dev [4].

Questions:

  1. Does anyone know how to formulate a virtual ALSA device that I could map the microphone array into structured input channels? I’ve found how the asound.conf [5] that seem to be mounting the device files to the /tmp directory, but I’m not yet familiar with ALSA to know how to extend it to what it may need.
  2. What are the microphone array extrinsics? Say the polar/euclidean coordinates and numeric mapping of each microphone in the array on the PCB layout reference frame? I tried looking for the metric measurements from any cad files, but I believe those are not open to the public.
  3. I’m assuming the PDM microphone array is synchronous, as the synchronous sampling of all channels as this is mandatory for the algorithm to work since it is based on time delays on pairs of microphones. But I’d like to double check with the creators, as I’m not sure where to find the source for the FPGA firmware to inspect how the array is sampled.

Links

[1] ManyEars / ManyEars-Wiki / Main_Page
GitHub - introlab/manyears: ManyEars Sound Source Localization, Tracking and Separation
[2] introlab/introlab-ros-pkg @ GitHub
[3] Adding back some C flags for arm on Pi · CogRob/manyears@8920983 · GitHub
[4] ManyEars / ManyEars-Wiki / UbuntuCompilation
[5] matrix-creator-init/asound.conf at 6dedc9c33681182873158c1aaadef44ca183c6f8 · matrix-io/matrix-creator-init · GitHub

1 Like

Hello!

I’m trying to do the same.

I think that Jack can do this task, but i’m experienced in neither Jack nor ManyEars.

I’ve installed and i’m successfully running Jack via the outlined guidelines here
http://wiki.linuxaudio.org/wiki/raspberrypi

I built ManyEars with the references to the Jack Libs

aplay- l is showing me a device with 8 inputs, i think?

**** List of PLAYBACK Hardware Devices ****
card 0: ALSA [bcm2835 ALSA], device 0: bcm2835 ALSA [bcm2835 ALSA]
Subdevices: 7/8
Subdevice #0: subdevice #0
Subdevice #1: subdevice #1
Subdevice #2: subdevice #2
Subdevice #3: subdevice #3
Subdevice #4: subdevice #4
Subdevice #5: subdevice #5
Subdevice #6: subdevice #6
Subdevice #7: subdevice #7
card 0: ALSA [bcm2835 ALSA], device 1: bcm2835 ALSA [bcm2835 IEC958/HDMI]
Subdevices: 1/1
Subdevice #0: subdevice #0
card 1: Dummy [Dummy], device 0: Dummy PCM [Dummy PCM]
Subdevices: 8/8
Subdevice #0: subdevice #0
Subdevice #1: subdevice #1
Subdevice #2: subdevice #2
Subdevice #3: subdevice #3
Subdevice #4: subdevice #4
Subdevice #5: subdevice #5
Subdevice #6: subdevice #6
Subdevice #7: subdevice #7

AND ManyEars can’t deal with it currently. I think there’s a step in Jack that I don’t understand?

Interested to see if you can take my work and push it across the finish line

Good luck!

Hi,

Did you make it work? I’m trying to do the same here.

+1
I also need the mic array as real device to use them with pulseaudio.
Please post here if someone has figured out how to do this.

Looks like there has been some activity to support proper linux audio, as mentioned by Kevin here:

2 Likes