Hi,

1. I have 3 4K-Cams with 2 streams each (one full res for recording, one low res for detection), run Frigate + Home Assistant on my board and it uses around 4GB memory. docker stats --no-stream reports that Frigate uses around 2GB of memory. Home Assistant uses around 300MB. I guess 4GB could work, but it is close to the limit. You could add some swap, it shouldn't slow down your system too much if you use a M.2 SSD. But keep in mind that you can't upgrade the memory, if you want to use your board for other stuff later that requires more. For an Orange Pi 5 Plus the surcharge for additional memory is rather low 4GB: $108, 8GB: $129 and 16GB: $151... So I would definitely buy 8GB or 16GB.
2. I would say that Frigate detects a person within 1 second after being almost fully visible in the camera stream. However, in my case the camera stream is some seconds delayed compared to the real world. I am not sure if this delay comes from my camera or if it comes somewhere from the software. Also, I don't know how long it takes until Home Assistant gets the event. But I would say that it is not possible to reliably trigger events within one second after they happen in the real world. The camera buffers the video, streams it to frigate, frigate (or rather ffmpeg) buffers the video, then there are additional processing and detection steps, some additional delay and buffering... I don' know how much one can optimize everything to have as low delay as possible. Maybe @NickM-27 has some info here.

Thank you for your reply. I have tested before, triggering the frigate/<camera_name>/motion event is quite fast, and Home Assistant (HA) will receive the information very quickly. I haven’t done precise testing, but I feel it’s within a second. However, compared to the motion event, I feel that the object event will be much slower. So I previously thought that as long as the object detection is faster, the total time might still be within a second, then it can barely cooperate with other smart devices. But now that I think about it, as you said, there are other factors that affect it, making it difficult to respond reliably within a second.

Hey @blakeblackshear,
do you think it is worth supporting Frigate+ for the rknn detector?

One feature of the Rockchip platform is "Model Encryption" that could help to prevent license violation. However, I haven't used it so far and don't know how it works. But I thought it might be interesting.

Since the model is trained on 640x640, what does changing the input_size to 320x320 actually do?

I'm considering adding YOLO-NAS as an available architecture for Frigate+ models. The models would then be compiled to ONNX format and that would enable broad support. Since they would be trained from scratch on a custom dataset, I can use them commercially.

Also, that export script in the README is a little different than the one on this page of the docs: https://docs.deci.ai/super-gradients/latest/documentation/source/models_export.html

I am not quite sure how pytorch adjusts the model to different input sizes. Here is the visualization of the first few layers of yolo-nas s 640x640 (left) and 320x320 (right):

I didn't notice any difference comparing both models (however, I just tested detection on a few images).

Also, that export script in the README is a little different than the one on this page of the docs: https://docs.deci.ai/super-gradients/latest/documentation/source/models_export.html

I wasn't aware of this function. I will have a look how it works and if the exported models are better.

I'm not sure either. I would guess that specifying a size other than the size from training would just add a step that upscales the data to the original input, but I don't really know.

Thanks! Adding a labelmap fixed the issue with the cat/dog. I've only seen the cat/people issue once or twice. Appreciate the help!

Hey @NickM-27,
I have a question in order to fix the issue mentioned above:
Is it safe to change config.model.labelmap_path from within the rknn.pys init function? Similar to how I set config.model.model_type here:

Or will this not work (or even result in a race condition, where the labelmap is sometimes loaded before and sometimes after the change)? Thank you!

See

I think it should be okay

Im not sure I would do that programmatically. Probably best to put it in the docs and have users reference a different labelmap path in their config. I am looking at YOLONAS for Frigate+, so I wouldn't recommend baking an opinionated approach to the labelmap in the code.

I would only set this, if the user uses the rknn model that comes with frigate. If the user provides his own model, the default value labelmap_path: /labelmap.txt will be still set and the user is able to override it. Also, I will just override the labelmap_path, if the default value /labelmap.txt is set. So, if the user provides his own labelmap_path: /config/my_labelmap.txt I won't override it, even if the user uses e.g. the yolonas.rknn model that comes with frigate.

Well, my 16gb system runs multiple days without this issue. In my setup, it occurred just a few times. But my setup uses around 4GB and rarely more. I think if you run more stuff that requires more memory, you will likely run into this issue more often.

Also, in order to reduce memory usage, you can set scale_rkrga=async_depth=0 as preset (see here. But this won't fix the problem.

At first, I figured this was the causing my issues with recordings, but then I found "Problem in 0.14.0 beta1... frigate stop recording every hour....needds to restart.. #11642". However, even if you aren't using more than 4GB, I would suspect that there is nothing telling the kernel to only use the first 4GB. It would seem that until they can fix the issue, it would be good if there is some kernel setting that would force it to only recognize the first 4GB. Do you know if this exists, or is my thinking flawed?

AFAIK, mpp requests an address within the first 4gb. The crash only happens, when there is no more space left.

Yes, go2rtc doesn't support hardware acceleration for the FFmpeg fork used in frigate. See here for more information AlexxIT/go2rtc#979

However, you can use go2rtc's exec command like this:

go2rtc:
  rtsp:
  streams:
      MyCam: exec:ffmpeg -hide_banner -allowed_media_types video+audio -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -i rtsp://user:pass@xxx.xxx.xxx.xxx:554/MainStream -c:v h264_rkmpp -g 50 -bf 0 -an -rtsp_transport tcp -f rtsp {{output}}

This transcodes h265 to h264 using hw accel and streams it to go2rtc.

Thanks for that! That should work with mjpeg as input stream as soon as their ffmpeg fork supports mjpeg decoding via mpp. Looks like that's on their to-do list.

Maybe there is a memory leak in the rknpu library. I haven't noticed this before but I will check if I can reproduce the issue.

Hey, thank you for posting your specs!

VPU (hardware video processing) working: no

May I help you to get HW acceleration running? I guess it will work in the new frigate v14.

Sure, I would like to try to get it working. With ghcr.io/blakeblackshear/frigate:0.14.0-rc2-rk I get

frigate  | 2024-08-05 11:30:56.935396179  [2024-08-05 11:30:56] detector.rknn                  INFO    : Starting detection process: 403
frigate  | 2024-08-05 11:30:56.936454339  Process detector:rknn:
frigate  | 2024-08-05 11:30:56.945069244  Traceback (most recent call last):
frigate  | 2024-08-05 11:30:56.945455991    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
frigate  | 2024-08-05 11:30:56.945468824      self.run()
frigate  | 2024-08-05 11:30:56.945608824    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
frigate  | 2024-08-05 11:30:56.945616990      self._target(*self._args, **self._kwargs)
frigate  | 2024-08-05 11:30:56.945720240    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
frigate  | 2024-08-05 11:30:56.945726948      object_detector = LocalObjectDetector(detector_config=detector_config)
frigate  | 2024-08-05 11:30:56.945817947    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
frigate  | 2024-08-05 11:30:56.945822906      self.detect_api = create_detector(detector_config)
frigate  | 2024-08-05 11:30:56.945907780    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
frigate  | 2024-08-05 11:30:56.945912155      return api(detector_config)
frigate  | 2024-08-05 11:30:56.945995571    File "/opt/frigate/frigate/detectors/plugins/rknn.py", line 39, in __init__
frigate  | 2024-08-05 11:30:56.946001404      model_props = self.parse_model_input(config.model.path, soc)
frigate  | 2024-08-05 11:30:56.946116320    File "/opt/frigate/frigate/detectors/plugins/rknn.py", line 83, in parse_model_input
frigate  | 2024-08-05 11:30:56.946120404      if "/" in model_path:
frigate  | 2024-08-05 11:30:56.946634901  TypeError: argument of type 'NoneType' is not iterable
frigate  | 2024-08-05 11:30:56.947398771  Exception ignored in: <function Rknn.__del__ at 0xffff89179d30>
frigate  | 2024-08-05 11:30:56.947546353  Traceback (most recent call last):
frigate  | 2024-08-05 11:30:56.947659519    File "/opt/frigate/frigate/detectors/plugins/rknn.py", line 62, in __del__
frigate  | 2024-08-05 11:30:56.949162760      self.rknn.release()
frigate  | 2024-08-05 11:30:56.949484174  AttributeError: 'Rknn' object has no attribute 'rknn'

Could you post your config.yml?

Do you have something like this in it:

model:
  path:

Delete the empty path line or specify a correct model like: path: deci-fp16-yolonas_m

Here is my config after adding a path element. I didn't have it at all before.

detectors:
  rknn:
    type: rknn
model:
  input_pixel_format: bgr
  path: deci-fp16-yolonas_m
objects:
  track:
    - person
    - car
    - dog
    - cat
    - bicyle
    - motorcycle
    - airplane
    - bus
mqtt:
  enabled: false
record:
  enabled: true
  events:
    retain:
      mode: active_objects
ffmpeg:
  hwaccel_args: preset-rk-h264 # or preset-rk-h265, depends on your cam
cameras:
  porch:
    ffmpeg:
      inputs:
        - path: rtsp://user:password@server.example.com/some-cam
          roles:
            - detect
            - record
version: 0.14

Frigate runs now, and pgrep -al ffmpeg shows that ffmpeg is running with -hwaccel rkmpp, -hwaccel_output_format drm_prime, and -vf scale_rkrga. In top the CPU is between 81% and 85% idle.

$ sudo cat /sys/kernel/debug/rkrga/load
num of scheduler = 1
================= load ==================
scheduler[0]: rga2
	 load = 7%
-----------------------------------
	 process 20: pid = 249918, name: ffmpeg [many parameters...]

Yes you are right there is an issue. Normally Frigate should default to deci-fp16-yolonas_n if there is no path specifies, but it doesn't. I will fix this. Thank you for reporting.

@MarcA711 Hi Marc, thanks for escalating this issue. I tested on the v14 rc2 - same result, reported here. It doesn't seem to be the problem related to the frigate version. Most likely the problem is between the RKNN toolkit and the model as suggested here.

Most likely the problem is between the RKNN toolkit and the model as suggested rockchip-linux/rknn-toolkit2#320.

Yes, this is also what I wrote above: It seems that there is an incompatibility between the yolov8 models from Frigate v13 and newer kernels.

However, upgrading frigate will also upgrade to newer models and eventually solve the problem. In your case, you have an old kernel with old rknn driver (v0.9.3). You should definitely upgrade to a newer version. I recommend this os: https://github.com/Joshua-Riek/ubuntu-rockchip

@cluangar I browsed the link but im lost. Any pointers for how to do this? I would like to specifically add chicken and egg detection. I have a cam out in my chicken coop and want to know when and how many eggs are layed. Only a few image sets contain eggs and they are very large, like ImageNet, so if I trained on a dataset like that I take it the model would run significantly slower?

I created example train.py . It's not good but work.
train_example.zip

Next step. I read from #14292

After success convert model to rknn. You must modify number classes in detection_api.py

I created yolonas convert tools. It's modify from DeepStream convertor and remove post process from onnx model. https://github.com/cluangar/Convertion-Tool-YOLO-NAS-to-RKNN

I will give this a try later on my Frigate+ model and see how it runs

Great. I will test on beta1-rk. but I modify rknn.py with support my custom model because my output only 8 classes not 80.

Bummer, no luck yet, seems the decoding interfaces are detected, but frigate:stable doesn't understand the rkmpp decoder:

error | 2025-03-17 21:20:17 | ffmpeg.cameraname.detect | Unrecognized hwaccel: rkmpp.
error | 2025-03-17 21:20:17 | ffmpeg.cameraname.detect | Supported hwaccels: vdpau vaapi drm opencl

Would it be possibly to just use one of the onnx2rknn converter scripts on these to make the use the NPU?

No, because currently the implementation expects and uses onnxruntime, not the detector specific plugins. This would require a pretty big overhaul to make it support rknn

Ok, thanks for the quick answer. Saved me a lot of time messing around

Thank you really much for providing these results!
Onvif should work in this branch as-well, the rockchip integration has nothing to do with it. However, I haven't tried it with my cameras. If you find something out, it would be nice if you can report it here!

Can you set parameters/ options how to export the model to onnx?

@MarcA711 Any progress on yolov8?:) Or what would you recommend for RKNN, yolonas? Basically, I have a cam in my chicken coop and would like to set up notifications upon eggs being detected through frigate+HASS.

yolonas does not perform as well on rknn due to the built in nms. yolov9 would be another option that is already supported for other detectors (in 0.16)

https://github.com/airockchip/yolov7 also looks like it would work

Is there any other option then yolov 7 and 9?

Can you set parameters/ options how to export the model to onnx?

I dont understand what you mean...

I pull the latest just about every other day if there seems to be commits/fixes related to my use case. Currently running image frigate:9e666dd-rk - Going to the face recognition page presents an empty/black screen that hangs for many minutes. Sometimes images will eventually load but then just clicking on the "+" button to try to associate a face with a particular person hangs for 30+ seconds before a popup even appears to select the persons name. Then another 60+ seconds before the training occurs. So very difficult to use anymore

how many images does it say are in the train tab

19396:) Do I need to clear manually?

the max is 100, you should delete them. Also, you need to check your config and make sure save_attempts is either not set or set to 100

thanks, that worked!

thank you for your hard work into this and sharing your findings
ive pull new dev image c91c697-rk and updated my yaml config
its running now :-)

i have an orange pi 3b laying around, it will be interesting how these new trained modells behave on npu 1 core
I only have 1 camera setup with frigate, so im trying to see if i can offload it to the pi 3b
I very much doubt it though. i might use the pi 3b with immich photo gallery and machine learning

im impressed though with rk3588 for such as cheap sbc

t will be the fastest and it will just get slower from there. I would start with t, it worked great in my testing

sorry one last thing, I cannot access the system#general for cpu metric anymore
the ui loads blank
everything else worths fine, but when i click "system metrics" nothing appears

im passing this in docker

and my yaml has
ffmpeg:
hwaccel_args: preset-rk-h264

is there something im doing wrong, it was working fine on stable-rk image with deci-fp16-yolonas_m

maybe the dev image build is the issue ?
ive tried the one in your screenshot [4c749e3-rk]

I get the same problem i cannot access the system metric ui

This is a bug that will be fixed, but if you add - /sys/:/sys/:ro to volumes then it will work now and include the NPU and GPU in the UI stats

thank you so much, its now working perfectly

@NickM-27 Just a heads up on the NPU stats. It looks like the code fails on a single NPU core Rockchip in the same way as rknputop which only works on multi core rockchips.

On a multicore RK

sudo cat /sys/kernel/debug/rknpu/load
NPU load:  Core0:  0%, Core1:  0%, Core2:  0%,

on a single core RK such as the 3566 it looks like this:

sudo cat /sys/kernel/debug/rknpu/load
NPU load:  9%

On a single NPU core RK, this line 80a13e4#diff-250250852a11e90c84e5950445c3dfed7892c64b4b7529040d07f57fa08100deR390
needs a check for the string 'NPU load: Core' else something like:

core_loads = re.findall(r"NPU load:\s+(\d+)%", npu_output)

Im halfway through a pull request to update rknputop to support single core RK, but my python skills are pretty ordinary so its taking some time.....

edit: just to confirm I have - /sys/:/sys/:ro in compose.yml

edit edit:
My Docker skills are pretty ordinary also, so I just edited frigate/util/services.py to my suggestion above and it works like a treat.

Thanks for the hint. I will definitely update the toolkit version soon.

so what can I do if I want to use my model?

I'm not sure if it is possible to remove the post processing steps after exporting to onnx, that's why the Frigate+ docs don't say that rknn is supported.

IS There any other way how i can Train my model?

Not directly, you could try emailing blake@frigate.video and see if the intermediate weights are saved then it could be exported not in the flat format

I dont mean via frigate+, in docs they say "You can also provide your own .rknn model" so there must be a way....