Ok, I will try this afternoon
On 1.34 I'm using same settings without issues.
memory leak in mocord
Re: memory leak in mocord
On 1.34 I'm using same settings without issues.
Production: zm 1.34.26 - Debian 11 | Test: zm 1.36.33 - Debian 12
Re: memory leak in mocord
lowered frame analysis to 5 and slowly, after 1 hour, memory leak happened again
Production: zm 1.34.26 - Debian 11 | Test: zm 1.36.33 - Debian 12
I have identified the problem
I need the recording to be divided into 30 second sections because I have to post-analyze the videos that have alarmed frames. With this setting the memory filling problem occurs.
With the default parameters instead it seems that the system is stable but it is not suitable for my needs.
As I wrote earlier with version 1.34.X it doesn't happen
- not-working.png (4.92 KiB) Viewed 11983 times
With the default parameters instead it seems that the system is stable but it is not suitable for my needs.
- working.png (5.38 KiB) Viewed 11983 times
As I wrote earlier with version 1.34.X it doesn't happen
Production: zm 1.34.26 - Debian 11 | Test: zm 1.36.33 - Debian 12
Re: memory leak in mocord
1.34 used a fixed length array to hold images. If it ran out it would log lots of errors but not consume ram.
1.36+ uses a non-fixed length queue. It is limited by MaxImageBuffers. It includes the encoded 264 data as well as decoded images so can use more ram than in 1.34. Again though, if you hit MaxImageBuffers packets in the queue, it will be logging all kinds of warnings and errors.
If you are running out of ram, ZM will be complaining about it bitterly. Go look in your logs.
1.36+ uses a non-fixed length queue. It is limited by MaxImageBuffers. It includes the encoded 264 data as well as decoded images so can use more ram than in 1.34. Again though, if you hit MaxImageBuffers packets in the queue, it will be logging all kinds of warnings and errors.
If you are running out of ram, ZM will be complaining about it bitterly. Go look in your logs.
Re: memory leak in mocord
these are my logging settingsiconnor wrote: ↑Fri Jul 28, 2023 2:17 pm 1.34 used a fixed length array to hold images. If it ran out it would log lots of errors but not consume ram.
1.36+ uses a non-fixed length queue. It is limited by MaxImageBuffers. It includes the encoded 264 data as well as decoded images so can use more ram than in 1.34. Again though, if you hit MaxImageBuffers packets in the queue, it will be logging all kinds of warnings and errors.
If you are running out of ram, ZM will be complaining about it bitterly. Go look in your logs.
- log.png (29.74 KiB) Viewed 11976 times
"Found locked packet when trying to free up video packets. This basically means that decoding is not keeping up."
"You have set the max video packets in the queue to 45. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval 16 is larger than this setting."
Is there anyone who can try to reproduce the anomaly by setting the parameters like me?
"General" section
Function: Mocord
Analysis Enabled: true
"Misc" section
Section length 30
Minimum section lenght 15
Production: zm 1.34.26 - Debian 11 | Test: zm 1.36.33 - Debian 12
Re: memory leak in mocord
I didnt have any footage I cared to save so I went ahead and deleted and recreated the monitors, kind of inconvenient but now everything appears to be normal again, knock on wood, and has been for a little bit, i'm barely at 1.5 GB of RAM usage now. Hubspot partner agency
stock market courses
stock market courses
Last edited by revplant on Wed Aug 23, 2023 1:21 pm, edited 1 time in total.
-
jrtaylor71
- Posts: 16
- Joined: Mon Oct 10, 2022 3:15 pm
Re: memory leak in mocord
There is a memory leak in 1.36 and 1.37. It happens when zmc restarts on a failed rtsp connection. It looks like it releases some of the memory not all of it and each time it restarts its releases less memory. 1.36 takes longer than 1.37. I have 2 different systems that do the same thing.
CPU usage climbs as the memory is used up until it uses up the swap file and the server becomes unresponsive.
Here is the kernel dump info from zmc. This is on a clean install that was done last night on Ubuntu 22 with 1.36.33. The VM is configured with 8 CPU's and 16GB of memory. The only thing I took from the old system was the config.
This is the config for the camera that was offline. It is currently set to none but was set to modcord.
CPU usage climbs as the memory is used up until it uses up the swap file and the server becomes unresponsive.
Here is the kernel dump info from zmc. This is on a clean install that was done last night on Ubuntu 22 with 1.36.33. The VM is configured with 8 CPU's and 16GB of memory. The only thing I took from the old system was the config.
Code: Select all
Aug 2 08:54:10 zoneminder kernel: [43799.510494] zmc invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Aug 2 08:54:10 zoneminder kernel: [43799.510530] CPU: 5 PID: 1329 Comm: zmc Not tainted 5.15.0-78-generic #85-Ubuntu
Aug 2 08:54:10 zoneminder kernel: [43799.510534] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
Aug 2 08:54:10 zoneminder kernel: [43799.510542] Call Trace:
Aug 2 08:54:10 zoneminder kernel: [43799.510550] <TASK>
Aug 2 08:54:10 zoneminder kernel: [43799.510568] show_stack+0x52/0x5c
Aug 2 08:54:10 zoneminder kernel: [43799.510610] dump_stack_lvl+0x4a/0x63
Aug 2 08:54:10 zoneminder kernel: [43799.510635] dump_stack+0x10/0x16
Aug 2 08:54:10 zoneminder kernel: [43799.510637] dump_header+0x53/0x228
Aug 2 08:54:10 zoneminder kernel: [43799.510649] oom_kill_process.cold+0xb/0x10
Aug 2 08:54:10 zoneminder kernel: [43799.510652] out_of_memory+0x106/0x2e0
Aug 2 08:54:10 zoneminder kernel: [43799.510678] __alloc_pages_slowpath.constprop.0+0x9b7/0xa80
Aug 2 08:54:10 zoneminder kernel: [43799.510694] __alloc_pages+0x311/0x330
Aug 2 08:54:10 zoneminder kernel: [43799.510697] alloc_pages+0x9e/0x1e0
Aug 2 08:54:10 zoneminder kernel: [43799.510704] __page_cache_alloc+0x7e/0x90
Aug 2 08:54:10 zoneminder kernel: [43799.510707] pagecache_get_page+0x152/0x590
Aug 2 08:54:10 zoneminder kernel: [43799.510709] ? page_cache_ra_unbounded+0x166/0x210
Aug 2 08:54:10 zoneminder kernel: [43799.510717] filemap_fault+0x488/0xab0
Aug 2 08:54:10 zoneminder kernel: [43799.510719] ? filemap_map_pages+0x309/0x400
Aug 2 08:54:10 zoneminder kernel: [43799.510722] __do_fault+0x3c/0x120
Aug 2 08:54:10 zoneminder kernel: [43799.510733] do_read_fault+0xeb/0x160
Aug 2 08:54:10 zoneminder kernel: [43799.510736] do_fault+0xa0/0x2e0
Aug 2 08:54:10 zoneminder kernel: [43799.510738] handle_pte_fault+0x1cd/0x240
Aug 2 08:54:10 zoneminder kernel: [43799.510741] __handle_mm_fault+0x405/0x6f0
Aug 2 08:54:10 zoneminder kernel: [43799.510744] handle_mm_fault+0xd8/0x2c0
Aug 2 08:54:10 zoneminder kernel: [43799.510747] do_user_addr_fault+0x1c9/0x670
Aug 2 08:54:10 zoneminder kernel: [43799.510765] exc_page_fault+0x77/0x170
Aug 2 08:54:10 zoneminder kernel: [43799.510779] asm_exc_page_fault+0x27/0x30
Aug 2 08:54:10 zoneminder kernel: [43799.510786] RIP: 0033:0x7fec9c1bd549
Aug 2 08:54:10 zoneminder kernel: [43799.510838] Code: Unable to access opcode bytes at RIP 0x7fec9c1bd51f.
Aug 2 08:54:10 zoneminder kernel: [43799.510840] RSP: 002b:00007fffcbcb85f0 EFLAGS: 00010206
Aug 2 08:54:10 zoneminder kernel: [43799.510843] RAX: 0000000a329a783b RBX: 000056390823a200 RCX: 0000000000000000
Aug 2 08:54:10 zoneminder kernel: [43799.510845] RDX: 0000000000000000 RSI: 0000000a32907180 RDI: 0000000000000001
Aug 2 08:54:10 zoneminder kernel: [43799.510847] RBP: 00007fffcbcb8600 R08: 0000000000000000 R09: 0000000000000000
Aug 2 08:54:10 zoneminder kernel: [43799.510848] R10: 00007fffcbd6c080 R11: 00007fffcbd6c090 R12: 0000000000000000
Aug 2 08:54:10 zoneminder kernel: [43799.510849] R13: 00005639081e2840 R14: 0000000000000000 R15: 00007fec9c24c5de
Aug 2 08:54:10 zoneminder kernel: [43799.510852] </TASK>
Aug 2 08:54:10 zoneminder kernel: [43799.510857] Mem-Info:
Aug 2 08:54:10 zoneminder kernel: [43799.510864] active_anon:1669367 inactive_anon:2238551 isolated_anon:0
Aug 2 08:54:10 zoneminder kernel: [43799.510864] active_file:124 inactive_file:379 isolated_file:0
Aug 2 08:54:10 zoneminder kernel: [43799.510864] unevictable:65969 dirty:0 writeback:0
Aug 2 08:54:10 zoneminder kernel: [43799.510864] slab_reclaimable:26330 slab_unreclaimable:20488
Aug 2 08:54:10 zoneminder kernel: [43799.510864] mapped:61343 shmem:61051 pagetables:12876 bounce:0
Aug 2 08:54:10 zoneminder kernel: [43799.510864] kernel_misc_reclaimable:0
Aug 2 08:54:10 zoneminder kernel: [43799.510864] free:33654 free_pcp:508 free_cma:0
Aug 2 08:54:10 zoneminder kernel: [43799.510871] Node 0 active_anon:6677468kB inactive_anon:8954204kB active_file:496kB inactive_file:1516kB unevictable:263876kB isolated(anon):0kB isolated(file):0kB mapped:245372kB dirty:0kB writeback:0kB shmem:244204kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 81920kB writeback_tmp:0kB kernel_stack:6800kB pagetables:51504kB all_unreclaimable? yes
Aug 2 08:54:10 zoneminder kernel: [43799.510876] Node 0 DMA free:13312kB min:60kB low:72kB high:84kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Aug 2 08:54:10 zoneminder kernel: [43799.510882] lowmem_reserve[]: 0 2886 15859 15859 15859
Aug 2 08:54:10 zoneminder kernel: [43799.510887] Node 0 DMA32 free:64232kB min:12288kB low:15360kB high:18432kB reserved_highatomic:0KB active_anon:1365508kB inactive_anon:1605768kB active_file:8kB inactive_file:4kB unevictable:0kB writepending:0kB present:3129280kB managed:3063744kB mlocked:0kB bounce:0kB free_pcp:1836kB local_pcp:496kB free_cma:0kB
Aug 2 08:54:10 zoneminder kernel: [43799.510892] lowmem_reserve[]: 0 0 12973 12973 12973
Aug 2 08:54:10 zoneminder kernel: [43799.510897] Node 0 Normal free:57072kB min:57276kB low:71080kB high:84884kB reserved_highatomic:2048KB active_anon:5311960kB inactive_anon:7348436kB active_file:1464kB inactive_file:1012kB unevictable:263876kB writepending:0kB present:13631488kB managed:13293204kB mlocked:263876kB bounce:0kB free_pcp:196kB local_pcp:0kB free_cma:0kB
Aug 2 08:54:10 zoneminder kernel: [43799.510902] lowmem_reserve[]: 0 0 0 0 0
Aug 2 08:54:10 zoneminder kernel: [43799.510906] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 2*2048kB (UM) 2*4096kB (M) = 13312kB
Aug 2 08:54:10 zoneminder kernel: [43799.510920] Node 0 DMA32: 723*4kB (UE) 757*8kB (UE) 517*16kB (UE) 318*32kB (UE) 204*64kB (UE) 68*128kB (UME) 20*256kB (UE) 4*512kB (ME) 6*1024kB (ME) 0*2048kB 0*4096kB = 62468kB
Aug 2 08:54:10 zoneminder kernel: [43799.510936] Node 0 Normal: 2162*4kB (UMEH) 1201*8kB (UMEH) 870*16kB (UMEH) 557*32kB (UME) 97*64kB (UMEH) 5*128kB (UMH) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56848kB
Aug 2 08:54:10 zoneminder kernel: [43799.510953] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Aug 2 08:54:10 zoneminder kernel: [43799.510955] 77956 total pagecache pages
Aug 2 08:54:10 zoneminder kernel: [43799.510960] 14012 pages in swap cache
Aug 2 08:54:10 zoneminder kernel: [43799.510964] Swap cache stats: add 2265765, delete 2251721, find 10994355/11078106
Aug 2 08:54:10 zoneminder kernel: [43799.510966] Free swap = 0kB
Aug 2 08:54:10 zoneminder kernel: [43799.510972] Total swap = 4194300kB
Aug 2 08:54:10 zoneminder kernel: [43799.510973] 4194190 pages RAM
Aug 2 08:54:10 zoneminder kernel: [43799.510974] 0 pages HighMem/MovableOnly
Aug 2 08:54:10 zoneminder kernel: [43799.510975] 101113 pages reserved
Aug 2 08:54:10 zoneminder kernel: [43799.510976] 0 pages hwpoisoned
Aug 2 08:54:10 zoneminder kernel: [43799.510976] Tasks state (memory values in pages):
Aug 2 08:54:10 zoneminder kernel: [43799.510977] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Aug 2 08:54:10 zoneminder kernel: [43799.510981] [ 446] 0 446 82425 608 684032 302 -250 systemd-journal
Aug 2 08:54:10 zoneminder kernel: [43799.510985] [ 493] 0 493 88721 6775 122880 0 -1000 multipathd
Aug 2 08:54:10 zoneminder kernel: [43799.510988] [ 496] 0 496 6664 512 81920 659 -1000 systemd-udevd
Aug 2 08:54:10 zoneminder kernel: [43799.510991] [ 710] 104 710 22338 462 73728 222 0 systemd-timesyn
Aug 2 08:54:10 zoneminder kernel: [43799.510994] [ 714] 0 714 12783 577 86016 494 0 VGAuthService
Aug 2 08:54:10 zoneminder kernel: [43799.510997] [ 715] 0 715 78753 685 110592 303 0 vmtoolsd
Aug 2 08:54:10 zoneminder kernel: [43799.510999] [ 814] 101 814 4029 711 69632 182 0 systemd-network
Aug 2 08:54:10 zoneminder kernel: [43799.511002] [ 816] 102 816 6382 367 94208 1000 0 systemd-resolve
Aug 2 08:54:10 zoneminder kernel: [43799.511004] [ 850] 0 850 1723 529 49152 48 0 cron
Aug 2 08:54:10 zoneminder kernel: [43799.511007] [ 851] 103 851 2195 635 53248 209 -900 dbus-daemon
Aug 2 08:54:10 zoneminder kernel: [43799.511009] [ 857] 0 857 20709 702 61440 65 0 irqbalance
Aug 2 08:54:10 zoneminder kernel: [43799.511017] [ 860] 0 860 8168 834 106496 2061 0 networkd-dispat
Aug 2 08:54:10 zoneminder kernel: [43799.511021] [ 862] 0 862 58624 285 86016 149 0 polkitd
Aug 2 08:54:10 zoneminder kernel: [43799.511024] [ 866] 107 866 55600 1051 86016 186 0 rsyslogd
Aug 2 08:54:10 zoneminder kernel: [43799.511026] [ 873] 0 873 3755 667 65536 157 0 systemd-logind
Aug 2 08:54:10 zoneminder kernel: [43799.511029] [ 875] 0 875 98143 735 122880 261 0 udisksd
Aug 2 08:54:10 zoneminder kernel: [43799.511032] [ 912] 0 912 1543 203 49152 6 0 agetty
Aug 2 08:54:10 zoneminder kernel: [43799.511034] [ 935] 0 935 3856 493 65536 375 -1000 sshd
Aug 2 08:54:10 zoneminder kernel: [43799.511037] [ 942] 0 942 27434 906 110592 1938 0 unattended-upgr
Aug 2 08:54:10 zoneminder kernel: [43799.511039] [ 954] 0 954 79489 746 118784 380 0 ModemManager
Aug 2 08:54:10 zoneminder kernel: [43799.511042] [ 1006] 114 1006 1311322 12201 991232 54216 0 mariadbd
Aug 2 08:54:10 zoneminder kernel: [43799.511045] [ 1026] 0 1026 61725 986 167936 1156 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511048] [ 1206] 33 1206 9762 1033 110592 2350 0 zmdc.pl
Aug 2 08:54:10 zoneminder kernel: [43799.511050] [ 1269] 33 1269 671442 517346 4919296 26746 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511053] [ 1291] 33 1291 867812 676961 6201344 47805 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511056] [ 1300] 33 1300 1785439 1191778 13762560 453550 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511058] [ 1313] 33 1313 166791 27139 925696 36238 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511061] [ 1329] 33 1329 257806 142527 1564672 10797 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511063] [ 1449] 33 1449 289326 153184 1822720 21341 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511066] [ 1492] 33 1492 14831 747 163840 8139 0 /usr/bin/zmcont
Aug 2 08:54:10 zoneminder kernel: [43799.511069] [ 1531] 33 1531 235912 60903 1134592 6016 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511072] [ 1608] 33 1608 13389 1635 143360 5825 0 zmfilter.pl
Aug 2 08:54:10 zoneminder kernel: [43799.511075] [ 1664] 33 1664 14580 2373 151552 6185 0 zmfilter.pl
Aug 2 08:54:10 zoneminder kernel: [43799.511077] [ 1672] 33 1672 16202 4811 172032 5309 0 zmaudit.pl
Aug 2 08:54:10 zoneminder kernel: [43799.511080] [ 1705] 33 1705 9627 1247 114688 2498 0 zmwatch.pl
Aug 2 08:54:10 zoneminder kernel: [43799.511082] [ 1735] 33 1735 9558 595 114688 2854 0 zmstats.pl
Aug 2 08:54:10 zoneminder kernel: [43799.511085] [ 13560] 33 13560 1030438 787912 7143424 44784 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511088] [ 13636] 33 13636 753024 339165 5419008 290108 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511091] [ 202410] 0 202410 385182 2422 339968 2040 -900 snapd
Aug 2 08:54:10 zoneminder kernel: [43799.511098] [ 395829] 0 395829 107671 5424 278528 6621 0 fwupd
Aug 2 08:54:10 zoneminder kernel: [43799.511101] [ 395908] 0 395908 59902 808 102400 107 0 upowerd
Aug 2 08:54:10 zoneminder kernel: [43799.511109] [ 396003] 0 396003 20346 529 57344 81 0 gpg-agent
Aug 2 08:54:10 zoneminder kernel: [43799.511112] [ 459049] 33 459049 61775 1272 196608 1422 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511115] [ 460976] 33 460976 62112 1257 200704 1515 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511117] [ 461401] 33 461401 62101 1752 196608 1169 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511120] [ 461592] 33 461592 62107 1072 200704 1707 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511122] [ 462408] 33 462408 62122 1242 196608 1580 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511125] [ 462581] 33 462581 48934 1313 176128 1181 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511128] [ 462582] 33 462582 62079 1059 196608 1577 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511130] [ 462653] 33 462653 62073 1628 196608 1065 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511133] [ 462770] 33 462770 62079 1560 196608 1094 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511135] [ 462771] 33 462771 62079 1624 196608 1102 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511138] [ 462854] 33 462854 62079 1530 196608 1145 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511140] [ 462919] 33 462919 62073 1502 196608 1107 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511143] [ 462929] 33 462929 15415 1091 118784 565 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511146] [ 462937] 33 462937 61889 907 139264 1120 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511148] [ 462939] 33 462939 15415 1069 118784 569 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511154] [ 462975] 33 462975 61811 1028 155648 1077 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511157] [ 462976] 33 462976 61889 796 139264 1130 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511159] [ 462977] 33 462977 60735 1632 188416 1014 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511163] [ 462979] 33 462979 15415 1159 118784 564 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511165] [ 462980] 33 462980 15415 1235 118784 564 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511168] [ 462982] 33 462982 61889 783 139264 1132 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511171] [ 462984] 33 462984 15415 1188 118784 569 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511173] [ 462985] 33 462985 15415 955 118784 566 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511176] [ 462986] 33 462986 15415 1102 118784 574 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511179] [ 463018] 33 463018 61891 807 139264 1123 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511181] [ 463023] 33 463023 62073 1329 196608 1067 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511184] [ 463060] 33 463060 61891 785 139264 1124 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511187] [ 463064] 33 463064 61891 763 139264 1125 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511193] [ 463065] 33 463065 15415 1019 118784 562 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511196] [ 463112] 33 463112 61811 1033 155648 1069 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511199] [ 463120] 33 463120 61873 745 139264 1119 0 apache2
Aug 2 08:54:10 zoneminder kernel: [43799.511202] [ 463147] 33 463147 56917 13896 339968 0 0 zmc
Aug 2 08:54:10 zoneminder kernel: [43799.511204] [ 463200] 0 463200 4058 1115 73728 0 0 sshd
Aug 2 08:54:10 zoneminder kernel: [43799.511207] [ 463203] 106 463203 3856 508 73728 2 0 sshd
Aug 2 08:54:10 zoneminder kernel: [43799.511209] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=zoneminder.service,mems_allowed=0,global_oom,task_memcg=/system.slice/zoneminder.service,task=zmc,pid=1300,uid=33
Aug 2 08:54:10 zoneminder kernel: [43799.511256] Out of memory: Killed process 1300 (zmc) total-vm:7141756kB, anon-rss:4741096kB, file-rss:1708kB, shmem-rss:24308kB, UID:33 pgtables:13440kB oom_score_adj:0
Code: Select all
*************************** 6. row ***************************
Id: 6
Name: SE-Corner
Notes:
ServerId: 0
StorageId: 2
Type: Ffmpeg
Function: None
Enabled: 1
DecodingEnabled: 1
LinkedMonitors: NULL
Triggers:
ONVIF_URL:
ONVIF_Username:
ONVIF_Password:
ONVIF_Options:
Device:
Channel: 0
Format: 0
V4LMultiBuffer: NULL
V4LCapturesPerFrame: 1
Protocol: NULL
Method: rtpRtsp
Host: NULL
Port:
SubPath:
Path: rtsp://192.168.173.61:554/11
SecondPath: NULL
Options: NULL
User: NULL
Pass: NULL
Width: 2560
Height: 1440
Colours: 4
Palette: 0
Orientation: ROTATE_0
Deinterlacing: 0
DecoderHWAccelName: NULL
DecoderHWAccelDevice: NULL
SaveJPEGs: 0
VideoWriter: 2
OutputCodec: 27
Encoder: libx264
OutputContainer: NULL
EncoderParameters: crf=1
RecordAudio: 0
RTSPDescribe: 0
Brightness: -1
Contrast: -1
Hue: -1
Colour: -1
EventPrefix: Event-
LabelFormat: %N - %d/%m/%y %H:%M:%S
LabelX: 0
LabelY: 0
LabelSize: 1
ImageBufferCount: 3
MaxImageBufferCount: 0
WarmupCount: 0
PreEventCount: 5
PostEventCount: 5
StreamReplayBuffer: 0
AlarmFrameCount: 1
SectionLength: 300
MinSectionLength: 10
FrameSkip: 0
MotionFrameSkip: 0
AnalysisFPSLimit: 2.00
AnalysisUpdateDelay: 0
MaxFPS: NULL
AlarmMaxFPS: NULL
FPSReportInterval: 100
RefBlendPerc: 6
AlarmRefBlendPerc: 6
Controllable: 0
ControlId: NULL
ControlDevice: NULL
ControlAddress: NULL
AutoStopTimeout: NULL
TrackMotion: 0
TrackDelay: NULL
ReturnLocation: -1
ReturnDelay: NULL
ModectDuringPTZ: 0
DefaultRate: 100
DefaultScale: 100
DefaultCodec: auto
SignalCheckPoints: 0
SignalCheckColour: #0000BE
WebColour: #e4fd16
Exif: 0
Sequence: 6
ZoneCount: 1
Refresh: NULL
Latitude: NULL
Longitude: NULL
RTSPServer: 0
RTSPStreamName:
Importance: Normal
Re: memory leak in mocord
That's totally possible, and should be easy to recreate.
I have several very flakey cameras though, and I don't see that here... but then I give my machine a lot of ram.
I have several very flakey cameras though, and I don't see that here... but then I give my machine a lot of ram.
-
jrtaylor71
- Posts: 16
- Joined: Mon Oct 10, 2022 3:15 pm
Re: memory leak in mocord
Here is what htop is showing. Monitor 10 is a camera that is on wifi that bounces on and off. It has lots of sub processes while all the rest only have 5 total.
Re: memory leak in mocord
a monitor should only have about 5 threads. Maybe a couple more ffmpeg is using a threaded decoder.
A possibility is that it is taking too long to close events. We use a separate thread for each event to do the writing. Closing an event should kill off the thread.
A possibility is that it is taking too long to close events. We use a separate thread for each event to do the writing. Closing an event should kill off the thread.
maybe I fixed it
I think my problem was not related to bad network or bad rstp stream.
On my test environment I had only 1 SSD disk (SO, database, video storage etc.) and I thought it was enough for 2 cameras
The behavior changed after I added a dedicated disk (mechanical) for video storage.
Now the memory consumption has remained stable after 1 hour. I've added more cameras, let's see what happens in the next few hours.
For now, thank you all.
On my test environment I had only 1 SSD disk (SO, database, video storage etc.) and I thought it was enough for 2 cameras
The behavior changed after I added a dedicated disk (mechanical) for video storage.
Now the memory consumption has remained stable after 1 hour. I've added more cameras, let's see what happens in the next few hours.
For now, thank you all.
Production: zm 1.34.26 - Debian 11 | Test: zm 1.36.33 - Debian 12