Hi there,
While using the Matrix Voice for a project, I have started to encounter kernel “oops” with the message “Unable to handle kernel paging request”, with an increased frequency as I used more system memory.
Apart from these crashes, everything is working as expected.
The crashes often take around 2-15 minutes to occur whilst my memory use is around 300MB. This continued to occur after a reinstall.
The dmesg output of one of these “oops” is below.
[ 227.970616] Unable to handle kernel paging request at virtual address 7f73b000
[ 227.974857] pgd = 80004000
[ 227.978914] [7f73b000] *pgd=35f88811, *pte=00000000, *ppte=00000000
[ 227.983268] Internal error: Oops: 807 [#1] SMP ARM
[ 227.987112] Modules linked in: veth xt_nat xt_tcpudp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay bnep hci_uart btbcm serdev bluetooth ecdh_generic matrixio_codec(O) matrixio_pcm(O) snd_soc_core matrixio_uart(O) snd_compress snd_pcm_dmaengine spidev evdev snd_usb_audio snd_hwdep snd_usbmidi_lib snd_rawmidi snd_seq_device brcmfmac brcmutil cfg80211 rfkill spi_bcm2835 fixed uio_pdrv_genirq uio bcm2835_v4l2(C) v4l2_common videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev media matrixio_env(O) matrixio_imu(O) industrialio matrixio_gpio(O) matrixio_everloop(O) matrixio_regmap(O) matrixio_core(O) snd_dummy
[ 228.004965] snd_pcm snd_timer snd ip_tables x_tables ipv6
[ 228.010286] CPU: 0 PID: 163 Comm: kworker/u8:2 Tainted: G C O 4.14.62-v7+ #1134
[ 228.015489] Hardware name: BCM2835
[ 228.020386] Workqueue: matrixio_pcm \xffffffa2ꪮ\xffffff8a\xffffffaa\xffffffba\xffffffaa\xffffffea\xffffffea\xffffffaa+\xffffffaa*\xffffffaa\xffffffaa\xffffffab+\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffae\xffffff8b\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffa2\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffff82ꪺ\xffffffae\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffab\xffffffab\xffffffaa*\xffffffaa\xffffffab\xffffffaa\xffffffaa\xffffffaa\xffffffab\xffffffa8\xffffffaa\xffffffaa\xffffffae\xffffffa8\xffffffaa\xffffffab\xffffffaa\xffffffaa\xffffffaa\xffffffba\xffffffa3\xfffffffa\xffffffaa\xffffffaa\xffffffaaꫪʪ\xffffffaa\xffffffaa\xffffffaa\xffffffaa讯\xffffffaa\xffffffaa\xffffffa9\xffffffaa\xffffffaa\xffffffaaꪪ\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffa8쪚\xffffffa8\xffffffa2\xffffffaa\xffffffaa\xffffffab\xffffffaa\xffffffaa\xffffffaa\xffffffaa*\xffffffaa*\xffffffae\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffbaⰪ [matrixio_pcm]
[ 228.025944] task: b6e03c00 task.stack: b6f00000
[ 228.031454] PC is at memcpy+0x244/0x330
[ 228.036675] LR is at 0x1d201e3
[ 228.042024] pc : [<80785984>] lr : [<01d201e3>] psr: 20000013
[ 228.047486] sp : b6f01e90 ip : 01e3022e fp : b6f01ed4
[ 228.052631] r10: 00000000 r9 : 0257024c r8 : 0231021c
[ 228.058038] r7 : 02100206 r6 : 01f201e0 r5 : 01cd01a8 r4 : 01a401f2
[ 228.063524] r3 : 02330219 r2 : 00000080 r1 : b88ce314 r0 : 7f73afe4
[ 228.068664] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 228.074103] Control: 10c5383d Table: 2d29406a DAC: 00000055
[ 228.079544] Process kworker/u8:2 (pid: 163, stack limit = 0xb6f00210)
[ 228.084873] Stack: (0xb6f01e90 to 0xb6f02000)
[ 228.090354] 1e80: 00000400 b6cf4618 7f73ad04 00000000
[ 228.096127] 1ea0: 00000088 7f73ad04 b6cf4610 7f0f915c 00000008 b6eb93b4 00003000 7f6f9d04
[ 228.101605] 1ec0: b6eb9398 00000088 b6f01efc b6f01ed8 7f6f7454 7f0f90e4 b6e1b880 b6eb93b4
[ 228.107301] 1ee0: b9c14400 b566fd00 00000000 00000088 b6f01f34 b6f01f00 801375b8 7f6f7400
[ 228.113108] 1f00: b9c14418 80c02d00 00000088 b9c14400 b6e1b898 b9c14400 b9c14418 80c02d00
[ 228.118179] 1f20: 00000088 b6e1b880 b6f01f7c b6f01f38 80137918 8013746c b6f01f5c b6f00000
[ 228.123470] 1f40: 00000000 80c02d00 80c882e2 b6f00038 b8ad4e9c b8ad4e80 00000000 b6e21400
[ 228.129205] 1f60: b6e1b880 801378b4 b8ad4e9c b9d0de80 b6f01fac b6f01f80 8013d99c 801378c0
[ 228.134690] 1f80: 40000700 b6e21400 8013d860 00000000 00000000 00000000 00000000 00000000
[ 228.140647] 1fa0: 00000000 b6f01fb0 8010810c 8013d86c 00000000 00000000 00000000 00000000
[ 228.146493] 1fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 228.151992] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[ 228.157789] [<80785984>] (memcpy) from [<7f0f915c>] (matrixio_read+0x84/0xa8 [matrixio_core])
[ 228.163640] [<7f0f915c>] (matrixio_read [matrixio_core]) from [<7f6f7454>] (\xffffffa2ꪮ\xffffff8a\xffffffaa\xffffffba\xffffffaa\xffffffea\xffffffea\xffffffaa+\xffffffaa*\xffffffaa\xffffffaa\xffffffab+\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffae\xffffff8b\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffa2\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffff82ꪺ\xffffffae\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffab\xffffffab\xffffffaa*\xffffffaa\xffffffab\xffffffaa\xffffffaa\xffffffaa\xffffffab\xffffffa8\xffffffaa\xffffffaa\xffffffae\xffffffa8\xffffffaa\xffffffab\xffffffaa\xffffffaa\xffffffaa\xffffffba\xffffffa3\xfffffffa\xffffffaa\xffffffaa\xffffffaaꫪʪ\xffffffaa\xffffffaa\xffffffaa\xffffffaa讯\xffffffaa\xffffffaa\xffffffa9\xffffffaa\xffffffaa\xffffffaaꪪ\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffa8쪚\xffffffa8\xffffffa2\xffffffaa\xffffffaa\xffffffab\xffffffaa\xffffffaa\xffffffaa\xffffffaa*\xffffffaa*\xffffffae\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffbaⰪ+0x7c50714c/0x7c507cf8 [matrixio_pcm])
[ 228.176048] [<7f6f7454>] (\xffffffa2ꪮ\xffffff8a\xffffffaa\xffffffba\xffffffaa\xffffffea\xffffffea\xffffffaa+\xffffffaa*\xffffffaa\xffffffaa\xffffffab+\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffae\xffffff8b\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffa2\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffff82ꪺ\xffffffae\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffab\xffffffab\xffffffaa*\xffffffaa\xffffffab\xffffffaa\xffffffaa\xffffffaa\xffffffab\xffffffa8\xffffffaa\xffffffaa\xffffffae\xffffffa8\xffffffaa\xffffffab\xffffffaa\xffffffaa\xffffffaa\xffffffba\xffffffa3\xfffffffa\xffffffaa\xffffffaa\xffffffaaꫪʪ\xffffffaa\xffffffaa\xffffffaa\xffffffaa讯\xffffffaa\xffffffaa\xffffffa9\xffffffaa\xffffffaa\xffffffaaꪪ\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffa8쪚\xffffffa8\xffffffa2\xffffffaa\xffffffaa\xffffffab\xffffffaa\xffffffaa\xffffffaa\xffffffaa*\xffffffaa*\xffffffae\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffaa\xffffffbaⰪ [matrixio_pcm]) from [<801375b8>] (process_one_work+0x158/0x454)
[ 228.182794] [<801375b8>] (process_one_work) from [<80137918>] (worker_thread+0x64/0x5b8)
[ 228.189388] [<80137918>] (worker_thread) from [<8013d99c>] (kthread+0x13c/0x16c)
[ 228.196295] [<8013d99c>] (kthread) from [<8010810c>] (ret_from_fork+0x14/0x28)
[ 228.202932] Code: e1a09829 e189980c e1a0c82c e18cc80e (e8a013f8)
[ 228.210445] ---[ end trace 2134a6e81a0f363d ]---
Unfortunately, my project is pretty all over the place as of now so sharing it is difficult and I have not been able to isolate the issues in an example program that also causes the crash. However, I will attempt to do so soon if I have some spare time.
For reference, my project interacts with the matrix voice via ODAS to track sources and via the provided python code for the everloop to provide feedback.
From my searches I could not seem to find anyone else who is encountering the issue regarding matrix devices, however some other people encountering similar causes for kernel "oops"s talk about the use of slub_debug
bootflag to aid debugging.
I have been testing this flag, and found it consistently triggers a kernel “oops” when performing an arecord
on a fresh install, which I believe could have the same cause as my original issue.
An example of the dmesg output from the above method is below.
[ 358.771969] Unable to handle kernel paging request at virtual address cccccd1c
[ 358.778146] pgd = a4af0000
[ 358.784070] [cccccd1c] *pgd=00000000
[ 358.790013] Internal error: Oops: 5 [#1] SMP ARM
[ 358.795992] Modules linked in: cmac bnep hci_uart btbcm serdev bluetooth ecdh_generic matrixio_codec(O) matrixio_pcm(O) spidev snd_soc_core snd_compress snd_pcm_dmaengine matrixio_uart(O) brcmfmac brcmutil cfg80211 rfkill spi_bcm2835 uio_pdrv_genirq uio fixed bcm2835_v4l2(C) v4l2_common videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_core videodev media matrixio_env(O) matrixio_imu(O) industrialio matrixio_gpio(O) matrixio_everloop(O) matrixio_regmap(O) matrixio_core(O) snd_dummy snd_bcm2835(C) snd_pcm snd_timer snd ip_tables x_tables ipv6
[ 358.817038] CPU: 0 PID: 1454 Comm: arecord Tainted: G C O 4.14.62-v7+ #1134
[ 358.824227] Hardware name: BCM2835
[ 358.829639] task: a4826cc0 task.stack: a4834000
[ 358.832976] PC is at soc_pcm_open+0x17c/0x8a0 [snd_soc_core]
[ 358.836116] LR is at proc_register+0xd4/0x150
[ 358.839340] pc : [<7f5f1670>] lr : [<802fc830>] psr: 60000013
[ 358.842630] sp : a4835c90 ip : a4835ae0 fp : a4835cf4
[ 358.845935] r10: 00000000 r9 : b939431c r8 : a511d640
[ 358.849234] r7 : 00000000 r6 : a5209708 r5 : a575de40 r4 : a5209100
[ 358.852448] r3 : cccccccc r2 : a499fb74 r1 : a499f96c r0 : 00000000
[ 358.855720] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 358.859065] Control: 10c5383d Table: 24af006a DAC: 00000055
[ 358.862425] Process arecord (pid: 1454, stack limit = 0xa4834210)
[ 358.865818] Stack: (0xa4835c90 to 0xa4836000)
[ 358.869100] 5c80: ffffffff 00000011 000f4240 00000009
[ 358.872501] 5ca0: a4b879c0 0000000d a4835cf4 a520914c a4b879c0 b93bd840 000f4240 00000011
[ 358.875983] 5cc0: 0000000b ffffffff a4835cfc 00000000 a4835d20 00000000 b9394308 00000001
[ 358.879494] 5ce0: b939431c 00000000 a4835d14 a4835cf8 7f0d0f18 7f5f1500 8079fb78 a575de40
[ 358.883046] 5d00: b9393f40 a4a3fb40 a4835d64 a4835d18 7f0d102c 7f0d0ebc a520ccc0 a4834018
[ 358.886439] 5d20: 00000003 00000000 a4826cc0 80148f58 b9394320 b9394320 a520d04c a4a3fb40
[ 358.889987] 5d40: b9393f40 a4a3fb40 a5330580 a51c9478 00000000 00000000 a4835d7c a4835d68
[ 358.893595] 5d60: 7f0d11d0 7f0d0f80 7f0d76b8 7f0ac300 a4835da4 a4835d80 7f0a15d4 7f0d1190
[ 358.897239] 5d80: 7f0a1524 7f0a937c 80d11538 b7945580 a51c9478 a4a3fb40 a4835dd4 a4835da8
[ 358.900886] 5da0: 8028f778 7f0a1530 a4a3fb48 00000058 a4a3fb40 a51c9478 a4a3fb48 8028f698
[ 358.904365] 5dc0: 00000000 a4835f5c a4835dfc a4835dd8 80286e20 8028f6a4 a4a3fb40 a4835ea8
[ 358.908016] 5de0: a4871280 a51c9478 00000000 a4835f5c a4835e1c a4835e00 802880cc 80286c2c
[ 358.911686] 5e00: a4826cc0 a4835ea8 00000000 00000802 a4835ea4 a4835e20 80299690 8028807c
[ 358.915343] 5e20: 80276d6c 80276364 60000013 8029a904 ba8d90c0 b9c008c0 00070001 a4834000
[ 358.918932] 5e40: a4835e64 a4835e50 00000041 a4a3fb40 00000000 00000006 00000000 00000002
[ 358.922543] 5e60: a51c9478 b9d08a50 b89c6008 7f0a55ac 40045532 7f0cb070 b909a100 00000004
[ 358.926252] 5e80: a4835f5c a4835ea8 00000001 80108204 a4834000 00000000 a4835f4c a4835ea8
[ 358.929988] 5ea0: 8029b670 802993d4 b9d08a50 b89c6008 7371274c 00000008 a52b6359 8029a904
[ 358.933742] 5ec0: 00000000 b9a8dab8 a51c9478 00000101 00000002 00000494 00000000 00000000
[ 358.937384] 5ee0: 00000000 a4835ee8 ffffff9c 00000004 b6e0e340 b909af40 00000000 00000400
[ 358.941194] 5f00: a4835f3c a4835f10 802ab0f0 802aa864 00080802 00080802 a52b6340 00000000
[ 358.945022] 5f20: 00000005 00000002 ffffff9c 00000000 00000004 ffffff9c a52b6340 00000005
[ 358.948872] 5f40: a4835f94 a4835f50 802884a8 8029b60c a4835fa4 a4a3f740 7ee0e14c 00000802
[ 358.952657] 5f60: 40040000 00000006 00000100 00000001 00080802 ffffffff 00000000 00000005
[ 358.956525] 5f80: 80108204 a4834000 a4835fa4 a4835f98 80288584 80288394 00000000 a4835fa8
[ 358.960460] 5fa0: 80108060 80288564 00080802 ffffffff 7ee0e29c 00080802 00404000 00404000
[ 358.964438] 5fc0: 00080802 ffffffff 00000000 00000005 76ef6394 7ee0e17c 7ee0e29c 00000001
[ 358.968400] 5fe0: 00000000 7ee0e154 76e642bc 76cd1c3c 60000010 7ee0e29c 00000000 00000000
[ 358.972385] [<7f5f1670>] (soc_pcm_open [snd_soc_core]) from [<7f0d0f18>] (snd_pcm_open_substream+0x68/0xc4 [snd_pcm])
[ 358.976543] [<7f0d0f18>] (snd_pcm_open_substream [snd_pcm]) from [<7f0d102c>] (snd_pcm_open+0xb8/0x210 [snd_pcm])
[ 358.980752] [<7f0d102c>] (snd_pcm_open [snd_pcm]) from [<7f0d11d0>] (snd_pcm_capture_open+0x4c/0x70 [snd_pcm])
[ 358.984954] [<7f0d11d0>] (snd_pcm_capture_open [snd_pcm]) from [<7f0a15d4>] (snd_open+0xb0/0x154 [snd])
[ 358.989069] [<7f0a15d4>] (snd_open [snd]) from [<8028f778>] (chrdev_open+0xe0/0x1a0)
[ 358.993328] [<8028f778>] (chrdev_open) from [<80286e20>] (do_dentry_open+0x200/0x328)
[ 358.997600] [<80286e20>] (do_dentry_open) from [<802880cc>] (vfs_open+0x5c/0x8c)
[ 359.001805] [<802880cc>] (vfs_open) from [<80299690>] (path_openat+0x2c8/0xff8)
[ 359.005949] [<80299690>] (path_openat) from [<8029b670>] (do_filp_open+0x70/0xd4)
[ 359.010235] [<8029b670>] (do_filp_open) from [<802884a8>] (do_sys_open+0x120/0x1d0)
[ 359.014551] [<802884a8>] (do_sys_open) from [<80288584>] (SyS_open+0x2c/0x30)
[ 359.018654] [<80288584>] (SyS_open) from [<80108060>] (ret_fast_syscall+0x0/0x28)
[ 359.022567] Code: e5943584 e15a0003 3affffe8 e5943008 (e5932050)
[ 359.026419] ---[ end trace dcd70b617fb8cbf2 ]---
To recreate this second crash with the flag, I performed the following actions:
-
apt-get update
,apt-get dist-upgrade
- Followed these instructions, installing
matrixio-malos
- Installed
matrixio-creator-init
,libmatrixio-creator-hal
,libmatrixio-creator-hal-dev
,matrixio-kernel-modules
, restarting inbetween as outlined here - Appended
slub_debug=FZ
to the end of the line of/boot/cmdline.txt
- Ran
arecord
and checked dmesg after
I am using a raspberry pi 3B+, however the error seems to occur on a Raspberry Pi 3B v1.2 aswell.
The outputs for the commands in the pinned post are as follows:
FPGA IDENTIFY = 6032bad2
FPGA VERSION = 1000a
gpiochip0: GPIOs 0-53, parent: platform/3f200000.gpio, pinctrl-bcm2835:
gpio-18 ( |sysfs ) out hi
gpio-24 ( |sysfs ) out hi
gpio-25 ( |sysfs ) out hi
gpio-26 ( |sysfs ) out hi
gpio-29 ( |led0 ) out lo
gpiochip1: GPIOs 128-135, brcmexp-gpio, can sleep:
gpio-130 ( |led1 ) out lo
gpiochip2: GPIOs 496-511, parent: spi/spi0.0, matrixio-gpio, can sleep:
Thanks for the help