TrueNAS Jail v1.34.21 - slow event deletion

[jonassimpson]

System specs:

Core i5-4690 CPU
32 GB RAM
6 x 8TB WD Red WD80EZZX-11C in Raidz2 pool

The box is very lightly used with almost no file service being used on the TrueNAS side (2 clients, less than one file access per day). It has six IP cameras doing 'mocord' at either 720p or 1080p, a couple via MJPEG and most via RTSP/ffmpeg.

I have a filter set to delete old events and to run in the background. It doesn't seem to run at all on its own and the disk space is filling up. When I login via SSH and run the filter manually it takes minutes to delete a single event (see log entries at bottom). When running and deleting events the console shows the load at between 1.2 and 1.6.

Any suggestions? Any way to test to see where the slowness is? There is no indication of a performance issue on the TrueNAS dashboard or anywhere else. Bottom line is that if it takes this long to delete events it will be a week or more to remove old entries, which seems ridiculous.

TIA,

JS

==========================
log entries below
==========================

07/05/2021 12:34:57.649989 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5360 from Monitor 7 StartTime:2021-05-31 07:02:52 from /var/db/zoneminder/events/7/2021-05-31/5360]
07/05/2021 12:37:39.094663 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5361 from Monitor 4 StartTime:2021-05-31 07:03:15 from /var/db/zoneminder/events/4/2021-05-31/5361]
07/05/2021 12:47:06.903134 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5362 from Monitor 3 StartTime:2021-05-31 07:03:45 from /var/db/zoneminder/events/3/2021-05-31/5362]
07/05/2021 12:47:15.574839 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5363 from Monitor 6 StartTime:2021-05-31 07:04:07 from /var/db/zoneminder/events/6/2021-05-31/5363]
07/05/2021 12:47:19.097888 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5364 from Monitor 1 StartTime:2021-05-31 07:04:38 from /var/db/zoneminder/events/1/2021-05-31/5364]
07/05/2021 12:58:44.240565 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5365 from Monitor 5 StartTime:2021-05-31 07:05:08 from /var/db/zoneminder/events/5/2021-05-31/5365]
07/05/2021 12:59:48.611995 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5366 from Monitor 2 StartTime:2021-05-31 07:11:16 from /var/db/zoneminder/events/2/2021-05-31/5366]
07/05/2021 13:01:19.171695 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5367 from Monitor 7 StartTime:2021-05-31 07:12:51 from /var/db/zoneminder/events/7/2021-05-31/5367]
07/05/2021 13:03:24.437689 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5368 from Monitor 4 StartTime:2021-05-31 07:13:09 from /var/db/zoneminder/events/4/2021-05-31/5368]
07/05/2021 13:11:55.950373 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5369 from Monitor 3 StartTime:2021-05-31 07:13:42 from /var/db/zoneminder/events/3/2021-05-31/5369]
07/05/2021 13:12:12.194326 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5370 from Monitor 6 StartTime:2021-05-31 07:14:07 from /var/db/zoneminder/events/6/2021-05-31/5370]
07/05/2021 13:12:14.120770 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5371 from Monitor 1 StartTime:2021-05-31 07:14:34 from /var/db/zoneminder/events/1/2021-05-31/5371]
07/05/2021 13:21:39.575386 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5372 from Monitor 5 StartTime:2021-05-31 07:15:08 from /var/db/zoneminder/events/5/2021-05-31/5372]
07/05/2021 13:22:36.388507 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5373 from Monitor 2 StartTime:2021-05-31 07:21:11 from /var/db/zoneminder/events/2/2021-05-31/5373]
07/05/2021 13:23:44.157878 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5374 from Monitor 7 StartTime:2021-05-31 07:22:49 from /var/db/zoneminder/events/7/2021-05-31/5374]
07/05/2021 13:25:23.990167 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5375 from Monitor 4 StartTime:2021-05-31 07:23:04 from /var/db/zoneminder/events/4/2021-05-31/5375]
07/05/2021 13:36:14.566424 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5376 from Monitor 6 StartTime:2021-05-31 07:23:29 from /var/db/zoneminder/events/6/2021-05-31/5376]
07/05/2021 13:36:21.454888 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5377 from Monitor 3 StartTime:2021-05-31 07:23:40 from /var/db/zoneminder/events/3/2021-05-31/5377]
07/05/2021 13:36:44.481171 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5378 from Monitor 1 StartTime:2021-05-31 07:24:31 from /var/db/zoneminder/events/1/2021-05-31/5378]
07/05/2021 13:45:28.317255 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5379 from Monitor 5 StartTime:2021-05-31 07:25:08 from /var/db/zoneminder/events/5/2021-05-31/5379]
07/05/2021 13:47:17.057524 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5380 from Monitor 2 StartTime:2021-05-31 07:31:06 from /var/db/zoneminder/events/2/2021-05-31/5380]
07/05/2021 13:48:16.829402 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5381 from Monitor 7 StartTime:2021-05-31 07:32:48 from /var/db/zoneminder/events/7/2021-05-31/5381]
07/05/2021 13:49:38.326227 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5382 from Monitor 4 StartTime:2021-05-31 07:33:01 from /var/db/zoneminder/events/4/2021-05-31/5382]
07/05/2021 14:01:50.539777 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5383 from Monitor 6 StartTime:2021-05-31 07:33:29 from /var/db/zoneminder/events/6/2021-05-31/5383]
07/05/2021 14:02:29.168475 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5384 from Monitor 3 StartTime:2021-05-31 07:33:37 from /var/db/zoneminder/events/3/2021-05-31/5384]
07/05/2021 14:03:01.906741 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5385 from Monitor 1 StartTime:2021-05-31 07:34:28 from /var/db/zoneminder/events/1/2021-05-31/5385]
07/05/2021 14:13:06.965457 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5386 from Monitor 5 StartTime:2021-05-31 07:35:08 from /var/db/zoneminder/events/5/2021-05-31/5386]
07/05/2021 14:13:37.224551 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5387 from Monitor 2 StartTime:2021-05-31 07:41:01 from /var/db/zoneminder/events/2/2021-05-31/5387]
07/05/2021 14:19:40.309091 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5388 from Monitor 6 StartTime:2021-05-31 07:41:16 from /var/db/zoneminder/events/6/2021-05-31/5388]
07/05/2021 14:20:11.138733 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5389 from Monitor 7 StartTime:2021-05-31 07:42:46 from /var/db/zoneminder/events/7/2021-05-31/5389]
07/05/2021 14:22:18.873983 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5390 from Monitor 4 StartTime:2021-05-31 07:42:59 from /var/db/zoneminder/events/4/2021-05-31/5390]
07/05/2021 14:35:22.977781 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5391 from Monitor 3 StartTime:2021-05-31 07:43:29 from /var/db/zoneminder/events/3/2021-05-31/5391]
07/05/2021 14:37:08.670216 zmfilter[3051].INF [ZoneMinder::Event:373] [Deleting event 5392 from Monitor 1 StartTime:2021-05-31 07:44:24 from /var/db/zoneminder/events/1/2021-05-31/5392]

[mikb]

In the filter settings: Have you accidentally ticked the "Only process [X] events" option to limit the number of events processed/deleted to *1* each time it runs?

What happens if you manually try and delete a single event from the drive (find the enclosing directory that has you .mp4 or collection of many many JPGs in, and "rm -r /nas/mount/path/to/event/" -- how long does it take to come back to you? Make sure you're blowing away the right directory when you do that, obviously!

I have seen occasions where my NAS suddenly slows down when accessing directories with a large number of entries, it's just crappy firmware to blame