Frigate + Coral TPU Install Troubleshooting

paulbok

I brought this up in the Home assistant thread, so am (finally) putting it in it's own discussion.

I bought a new PC earlier in the year to use as a server for a number of services, Plex, Home Assistant and in particular, Frigate for my cctv cameras. I got a Coral M.2. TPU to add image processing to Frigate but have been unable to get it to work. As of yet, i can't read the TPU so don't know if that is faulty or I'm just not installing correctly. The Linux world is fairly new to me so it's hopefully me just missing something on the setup.

A couple of you advised to make this thread to help, so thanks in advance. @Jonathan I'm not in Dublin so can't avail of your kind offer to test the TPU card.

Equipment.

HP Prodesk 600 G5 SFF. 8 core i7-9700, 32gb ram, 1tb ssd.

Running proxmox VE 8.2.4 as hypervisor.

Went with a Coral dual edge M.2 TPU to make use of that slot on the motherboard. Having read up on that PC model, it is supposed to allow passthrough of such devices.

What I've done so far to install;

Reading the available slot.

Handle 0x003C, DMI type 9, 17 bytes
System Slot Information
Designation:



Slot3 / M2 WLAN/BT
Type: x1 PCI Express 3 x1
Current Usage: Available
Length:

Other
ID: 3
Characteristics:
3.3 V is provided
PME signal is supported
SMBus signal is supported
Bus Address: 0000:00:1c.7

However, there is no mention of that slot from the lspci query.

Using the TPU install guide form the Coral home page

Getting this error on the first step, installing the Debian package repository

Err:8 https://packages.cloud.google.com/apt coral-cloud-stable Release
404 Not Found [IP: 216.58.201.110 443]
Reading package lists... Done
E: The repository 'https://packages.cloud.google.com/apt coral-cloud-stable Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: https://downloads.plex.tv/repo/deb/dists/public/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://packages.cloud.google.com/apt/dists/coral-edgetpu-stable/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

Jonathan

See page 14 of the attached. What slot is the coral in?

http://h10032.www1.hp.com/ctg/Manual/c06404289.pdf

Per lspci output above, slot 3 is only a single lane PCI slot, while the dual edge TPU requires two lanes (one per coral). I'm not clear if a single lane slot will allow one or none TPUs to function. There is some good info here:

https://github.com/magic-blue-smoke/Dual-Edge-TPU-Adapter

paulbok

I had to open it up to double check but it's definitely in No. 10. on that motherboard diagram, the M2 wlan slot.

I had read somewhere that it should have picked up one lane off the TPU if it didn't take both but obviously not.

Had a look at that adapter it's for a PCIe x 1 slot, so the below should all work together and will run both lanes. Haven't any other use for that PCIe slot anyway.

$37 including postage, likely to incur €3/4? import duty so maybe €40 in total.

Could just pick up a usb TPU for €75 and be done with it, but think I'll go with this adapter.

Will update once I get it.

Now, onto moving my Plex server to Proxmox while I'm waiting.

Jonathan

Welcome to stick it in padded envelope in the post and I'll test it for you to ensure that it isn't DOA before you go buying the adapter. Alternatively, it works on mine, I'm happy to swap you my single TPU if the dual PCIe lane is the issue. No benefit to me as my Optiplex 7060 only has a single PCIe lane but might mean we both have working TPUs then. Unless you have a load of cameras and need the second TPU?

paulbok

Thanks Jonathan, but I went ahead and ordered that adapter last night before seeing your post.

I will have a good few cameras when I'm finished, probably will have one or two around my sister's house next door when I'm done.

I see the chap making those adapters is planning a new one to carry two dual TPUs. Would be some serious imaging power if he can make it work.

paulbok

Finally got around to installing the TPU on the PCIE adapter and both lanes show up fine now.

04:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]


05:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]

Next issue is the PCIE drivers are not installing for me, neither apex nor gasket-dkms are showing up.

when check for them, I get nothing back.

In your PVE shell, run "lsmod | grep apex"  You should see the apex and gasket.. i.e:
root@pve:~# lsmod | grep apex
apex 28672 5
gasket 126976 6 apex

From reading up on it, there was a problem early in the year with support from Google for the TPU for the drivers, but I found some posts on how to rebuild the gasket-dkms module but it hasn't helped. Not seeing any specific errors when running those commands.
The only thing I did notice is, it tries to install to

Installing to /lib/modules/6.8.12-4-pve/updates/dkms/

but my kernel version is 6.8.12-3-pve, could that be a reason?

Jonathan

https://www.boards.ie/discussion/comment/122870799#Comment_122870799

Can you outline what steps you've tried to date?

As regards the different kernel version, likely you have installed a kernel update via apt, but you haven't rebooted yet, so you are still running on the old kernel.

paulbok

Kernel issue resolved on a reboot.

I'm trying to follow this guide for installing.
https://coral.ai/docs/m2/get-started/#2a-on-linux.

The first 2 steps are a per the instructions.

Next, installing the debian packages does throw up some errors, I think?

root@prox:~# echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list

deb https://packages.cloud.google.com/apt coral-edgetpu-stable main

root@prox:~# curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

Dload  Upload   Total   Spent    Left  Speed

0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).

100  1022  100  1022    0     0   6354      0 --:--:-- --:--:-- --:--:--  6387

OK

root@prox:~#

I don't know what this is telling me.

root@prox:~# sudo apt-get update

Hit:1 http://download.proxmox.com/debian/pve bookworm InRelease

Hit:2 http://ftp.ie.debian.org/debian bookworm InReleaseHit:3 https://downloads.plex.tv/repo/deb public InReleaseHit:4 http://security.debian.org bookworm-security InReleaseHit:5 http://ftp.ie.debian.org/debian bookworm-updates InReleaseIgn:6 https://packages.cloud.google.com/apt coral-cloud-stable InRelease

Hit:7 https://packages.cloud.google.com/apt coral-edgetpu-stable InRelease

Err:8 https://packages.cloud.google.com/apt coral-cloud-stable Release

404  Not Found [IP: 142.250.187.238 443]

Reading package lists... Done

E: The repository 'https://packages.cloud.google.com/apt coral-cloud-stable Release' no longer has a Release file.

N: Updating from such a repository can't be done securely, and is therefore disabled by default.

N: See apt-secure(8) manpage for repository creation and user configuration details.

W: https://downloads.plex.tv/repo/deb/dists/public/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

W: https://packages.cloud.google.com/apt/dists/coral-edgetpu-stable/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.

Installing the PCIe driver after that looks ok?

root@prox:~# sudo apt-get install gasket-dkms libedgetpu1-std

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

gasket-dkms is already the newest version (1.0-18).

libedgetpu1-std is already the newest version (16.0).

The following packages were automatically installed and are no longer required:

proxmox-kernel-6.8.12-2-pve-signed proxmox-kernel-6.8.8-2-pve-signed

Use 'sudo apt autoremove' to remove them.

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

User setup looks ok

groupadd: group 'apex' already exists

adduser: The user `root' is already a member of `apex'.

Then I reboot the container and check, TPU picked up fine, but PCIe driver isn't.

root@prox:~# lspci -nn | grep 089a

04:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]

05:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]

root@prox:~# ls /dev/apex_0

ls: cannot access '/dev/apex_0': No such file or directory

10-10-20

https://www.boards.ie/discussion/comment/122881583#Comment_122881583

pve-headers for Proxmox (if that's what the host is).

https://forum.proxmox.com/threads/install-of-pcie-drivers-for-coral-tpu.95503/#post-478721

But yeah - not sure why you're seeing a 404 on that repo page… maybe they moved it.

paulbok

Where should those drivers install?

I know SFA on linux file systems but having them in root like this doesn't look right?

Jonathan

Well based on the output above, you have installed the gasket-dkms package, which is good.

However, can you run the command below? This will confirm if you have built (via dkms) the two required kernel modules for the running kernel version. If you have, you'll see the two .ko files below.

root@host:~# ls -l /lib/modules/$(uname -r)/updates/dkms/

total 204

-rw-r--r-- 1 root root  42117 Sep 23 15:58 apex.ko

-rw-r--r-- 1 root root 161069 Sep 23 15:58 gasket.ko

And to be extra sure, can you post the output of the below too?

root@host:~# uname -r; dkms status
6.1.0-25-amd64

Deprecated feature: REMAKE_INITRD (/var/lib/dkms/gasket/1.0/source/dkms.conf)

Deprecated feature: REMAKE_INITRD (/var/lib/dkms/gasket/1.0/source/dkms.conf)

Deprecated feature: REMAKE_INITRD (/var/lib/dkms/gasket/1.0/source/dkms.conf)

gasket/1.0, 6.1.0-25-amd64, x86_64: installed

gasket/1.0, 6.1.0-27-amd64, x86_64: installed

gasket/1.0, 6.1.0-28-amd64, x86_64: installed

paulbok

They aren't built going by this.😫

root@prox:~# ls -l /lib/modules/$(uname -r)/updates/dkms/

ls: cannot access '/lib/modules/6.8.12-4-pve/updates/dkms/': No such file or directory


root@prox:~# uname -r; dkms status

6.8.12-4-pve

gasket/1.0: added

root@prox:~#

Jonathan

https://www.boards.ie/discussion/comment/122900461#Comment_122900461

Okay. Run the following then and post the output.

apt install pve-headers

dkms autoinstall

paulbok

first one

root@prox:~# apt install pve-headers

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

pve-headers is already the newest version (8.3.0).

The following packages were automatically installed and are no longer required:

proxmox-kernel-6.8.12-2-pve-signed proxmox-kernel-6.8.8-2-pve-signed

Use 'apt autoremove' to remove them.

0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.

1 not fully installed or removed.

After this operation, 0 B of additional disk space will be used.

Do you want to continue? [Y/n] y

Setting up gasket-dkms (1.0-18) ...

Removing old gasket-1.0 DKMS files...

Deleting module gasket-1.0 completely from the DKMS tree.

Loading new gasket-1.0 DKMS files...

Deprecated feature: REMAKE_INITRD (/usr/src/gasket-1.0/dkms.conf)

Building for 6.8.12-4-pve

Building initial module for 6.8.12-4-pve

Deprecated feature: REMAKE_INITRD (/var/lib/dkms/gasket/1.0/source/dkms.conf)

Error! Bad return status for module build on kernel: 6.8.12-4-pve (x86_64)

Consult /var/lib/dkms/gasket/1.0/build/make.log for more information.

dpkg: error processing package gasket-dkms (--configure):

installed gasket-dkms package post-installation script subprocess returned error exit status 10

Errors were encountered while processing:

gasket-dkms

E: Sub-process /usr/bin/dpkg returned an error code (1)

root@prox:~#

Second one

root@prox:~# dkms autoinstall

Deprecated feature: REMAKE_INITRD (/var/lib/dkms/gasket/1.0/source/dkms.conf)

Deprecated feature: REMAKE_INITRD (/etc/dkms/framework.conf)

Sign command: /lib/modules/6.8.12-4-pve/build/scripts/sign-file

Signing key: /var/lib/dkms/mok.key

Public certificate (MOK): /var/lib/dkms/mok.pub

Deprecated feature: REMAKE_INITRD (/var/lib/dkms/gasket/1.0/source/dkms.conf)


Building module:

Cleaning build area...

make -j8 KERNELRELEASE=6.8.12-4-pve -C /lib/modules/6.8.12-4-pve/build M=/var/lib/dkms/gasket/1.0/build...(bad exit status: 2)

Error! Bad return status for module build on kernel: 6.8.12-4-pve (x86_64)

Consult /var/lib/dkms/gasket/1.0/build/make.log for more information.

Error! One or more modules failed to install during autoinstall.

Refer to previous errors for more information.

root@prox:~# 

Jonathan

You'll have to build the latest version of the drivers from source then.

apt install dh-dkms devscripts git
git clone https://github.com/google/gasket-driver.git
cd gasket-driver
debuild -us -uc -tc -b -d
dpkg -i ../gasket-dkms_*.deb
reboot

Let me know how you get on. If you get stuck I'll build it here and host the .deb file somewhere.

paulbok

Thanks Jonathan, I'm away this weekend so could be Monday before I get time to try this.

paulbok

https://www.boards.ie/discussion/comment/122903348#Comment_122903348

got an error on that straight away, I wonder if I need to any additional repositories to find that package?

root@frigate-lxc:~# apt install dh-dkms devscripts git

Reading package lists... Done

Building dependency tree... Done

Reading state information... Done

E: Unable to locate package dh-dkms

root@frigate-lxc:~#