1.30->1.32->1.34 upgrade: multiple broken monitors

[marcmerlin]

I have been happily running zoneminder for almost 10 years, and upgraded to 1.30 without issues.
Then, I was forced to upgrade 1.32 due to some php dependency hell.
After that, many things broke, listed in details here:
viewtopic.php?f=38&t=28830
ultimately I was told that I shouldn't be running anything so old, but it was the last version available in debian, and compiling my own looked like another rat's nest I wanted to avoid.

Debian just added 1.34.8-1, so I upgraded to that.
And things are about just as broken, if not more.
I put logs from startup here:
https://gist.githubusercontent.com/marc ... tfile1.txt
lots of different errors of different kinds, the most pressing is that after working for a few seconds, all/most working monitors cycle through

Apr 11 17:08:48 gargamel zmtrigger[17364]: ERR [Memory map file '/dev/shm/zm.mmap.5' should have been 1160 but was instead 0]
Apr 11 17:08:48 gargamel zmtrigger[17364]: WAR [no memkey in zmMemInvalidate]
Apr 11 17:08:48 gargamel zmtrigger[17364]: ERR [Memory map file '/dev/shm/zm.mmap.5' should have been 1160 but was instead 0]
Apr 11 17:08:48 gargamel zmtrigger[17364]: ERR [Memory map file '/dev/shm/zm.mmap.8' does not exist. zmc might not be running.]
Apr 11 17:08:48 gargamel zmdc[16886]: INF ['zma -m 8' starting at 20/04/11 17:08:48, pid = 18381]

machine (32bit userland, 64bit kernel) has 40GB of RAM, and 16GB in /dev/shm/. Df shows that I'm not running out of RAM.

gargamel:/dev/shm# df -h .
Filesystem Size Used Avail Use% Mounted on
tmpfs 16G 4.4G 12G 28% /run/shm

What should I try next?
Thanks

[MoOz]

I suggest that you upgrade to the latest Ubuntu 20
worked 1st go
clean install
faster than trying to fix the upgrade

https://wiki.zoneminder.com/Ubuntu_Serv ... der_1.34.x

[marcmerlin]

Sorry, I'm confused. Are you saying that upgrades are known not to work and/or that debian is not supported, so I should delete my server with literally 20 years of history, wipe it, to install another distribution with a known history of lesser support (i.e. ubuntu)?
If so, that's a bit of a hard sell and makes me want to look into other solutions that are going to be better supported and not just randomly break when I upgrade.

That said, have you looked at
https://gist.githubusercontent.com/marc ... tfile1.txt ?
Can you confirm that the multiple errors are going to be fixed, including the ones where zm seems to have issues processing incoming video data from video cameras it could talk to perfectly fine in older versions?

[bbunge]

Think the recommendation to upgrade to a beta, Ubuntu 20.04 is a bit funny and unwise.

For Debian you should be using the builds in the zmrepo: https://zmrepo.zoneminder.com/debian/

[iconnor]

Apr 11 17:07:59 gargamel zmc_m2[16929]: WAR [zmc_m2] [Unknown method (simple)]

Tells me that this camera needs you to review it's settings. What type of monitor is it? Remote or FFMPEG?
I recommend using the ffmpeg input type unless you find you have problems with it.
A lot of the errors further down lead me to think our old remote method code can't handle something your camera is sending.

Also, turn off ZMAUDIT. Only run it after a crash if you need to clean things up.

Turn on debug for more information about the inability to capture.

[marcmerlin]

Apr 11 17:07:59 gargamel zmc_m2[16929]: WAR [zmc_m2] [Unknown method (simple)]
Tells me that this camera needs you to review it's settings. What type of monitor is it? Remote or FFMPEG?
I recommend using the ffmpeg input type unless you find you have problems with it.
A lot of the errors further down lead me to think our old remote method code can't handle something your camera is sending.
Also, turn off ZMAUDIT. Only run it after a crash if you need to clean things up.
Turn on debug for more information about the inability to capture.

Hi @iconnor, sorr yfor the very late answer, stuff happened and couldn't look at this for a while.

I just turned off zmaudit, that was probably forgotten from a while ago, thanks.

m2 is actually mostly working. Here are my top errors for 1h (sort | uniq -c | sort -rn ). "only" 6 cameras fail repeatedly
- 5: http simple /mjpeg/snap.cgi?chn=0
- 6: http simple /IMAGE.JPG
- 8: http simple /mjpeg/snap.cgi?chn=0
- 16: http simple /mjpeg/snap.cgi?chn=0
- 17: rtp/unicast /live/ch0
- 19: http simple /mjpeg/snap.cgi?chn=0

I'm dismayed, http get is like the simplest way to get video, no ffmpeg or h264 decoding issues, it just works, well it just worked for 10 years or so until I upgraded zoneminder one time too many (it was fine all the way to 1.30):

7486 zmtrigger[14396]: ERR [Memory map file '/dev/shm/zm.mmap.16' should have been 1160 but was instead 0]
7414 zmtrigger[14396]: ERR [Memory map file '/dev/shm/zm.mmap.19' should have been 1160 but was instead 0]
7404 zmtrigger[14396]: ERR [Memory map file '/dev/shm/zm.mmap.8' should have been 1160 but was instead 0]
7237 zmtrigger[14396]: ERR [Memory map file '/dev/shm/zm.mmap.5' should have been 1160 but was instead 0]
1816 zmtrigger[14396]: ERR [Memory map file '/dev/shm/zm.mmap.6' should have been 1160 but was instead 0]
1492 zmc_m23[23257]: WAR [zmc_m23] [Discarding packet for unrecognised ssrc 9fa23cef]
748 zmc_m23[23257]: WAR [zmc_m23] [Discarding frame 11]
402 zmdc[14143]: ERR ['zma -m 16' exited abnormally, exit status 255]
399 zmdc[14143]: ERR ['zma -m 8' exited abnormally, exit status 255]
398 zmdc[14143]: ERR ['zma -m 19' exited abnormally, exit status 255]
329 zmdc[14143]: ERR ['zma -m 5' exited abnormally, exit status 255]
275 zmdc[14143]: ERR ['zmc -m 17' exited abnormally, exit status 255]
263 zmc_m23[23257]: WAR [zmc_m23] [Packet out of sequence, gap 64705]
191 zmc_m23[23257]: WAR [zmc_m23] [Discarding frame 13]
143 zmwatch[14408]: WAR [no memkey in zmMemInvalidate]
139 zmtrigger[14396]: ERR [Memory map file '/dev/shm/zm.mmap.5' does not exist. zmc might not be running.]
136 zmdc[14143]: INF ['zmc -m 5' exited normally]
136 zmdc[14143]: INF ['zmc -m 19' exited normally]
136 zmdc[14143]: INF ['zmc -m 16' exited normally]
135 zmdc[14143]: INF ['zmc -m 8' exited normally]
130 zmc_m23[23257]: WAR [zmc_m23] [Packet in sequence, gap 832]
126 zmc_m23[23257]: WAR [zmc_m23] [Packet out of sequence, gap 64706]
118 zmdc[14143]: INF ['zma -m 5' exited, signal 14]
115 zmdc[14143]: INF ['zma -m 19' exited, signal 14]
114 zmtrigger[14396]: ERR [Memory map file '/dev/shm/zm.mmap.8' does not exist. zmc might not be running.]
112 zmdc[14143]: INF ['zma -m 8' exited, signal 14]
108 zmdc[14143]: INF ['zma -m 16' exited, signal 14]
108 zmc_m23[23257]: WAR [zmc_m23] [Packet out of sequence, gap 64704]
105 zmtrigger[14396]: ERR [Memory map file '/dev/shm/zm.mmap.16' does not exist. zmc might not be running.]

[marcmerlin]

For debug, I have LOG_LEVEL_SYSLOG to Info, is it enough to turn that to debug?

Or do I need LOG_DEBUG + LOG_DEBUG_TARGET + LOG_DEBUG_LEVEL?
If so, what target and what level?

the vexing thing is that I have other http/simple cameras that do work still.
It looks like the mmap code has issues and I can't tell what. I see that everything fits fine:

Code: Select all

-rw-------  1 www-data www-data  311045740 May  3 18:37 zm.mmap.1
-rw-------  1 www-data www-data  147461660 May  3 18:37 zm.mmap.10
-rw-------  1 www-data www-data  147461660 May  3 18:37 zm.mmap.11
-rw-------  1 www-data www-data  308741740 May  3 18:46 zm.mmap.16
-rw-------  1 www-data www-data  311045740 May  3 18:46 zm.mmap.17
-rw-------  1 www-data www-data  311045740 May  3 18:46 zm.mmap.19
-rw-------  1 www-data www-data  110597660 May  3 18:46 zm.mmap.2
-rw-------  1 www-data www-data 1105926140 May  3 18:37 zm.mmap.23
-rw-------  1 www-data www-data 1105926140 May  3 18:37 zm.mmap.24
-rw-------  1 www-data www-data   46085740 May  3 18:37 zm.mmap.4
-rw-------  1 www-data www-data  138245740 May  3 18:46 zm.mmap.5
-rw-------  1 www-data www-data   27653580 May  3 18:46 zm.mmap.6
-rw-------  1 www-data www-data  311045740 May  3 18:46 zm.mmap.8
-rw-------  1 www-data www-data  311045740 May  3 18:37 zm.mmap.9

and then a few seconds later, I see some that become 0

Code: Select all

-rw-------  1 www-data www-data  311045740 May  3 18:47 zm.mmap.1
-rw-------  1 www-data www-data  147461660 May  3 18:37 zm.mmap.10
-rw-------  1 www-data www-data  147461660 May  3 18:37 zm.mmap.11
-rw-------  1 www-data www-data  308741740 May  3 18:47 zm.mmap.16
-rw-------  1 www-data www-data  311045740 May  3 18:47 zm.mmap.17
-rw-------  1 www-data www-data          0 May  3 18:47 zm.mmap.19
-rw-------  1 www-data www-data  110597660 May  3 18:47 zm.mmap.2
-rw-------  1 www-data www-data 1105926140 May  3 18:37 zm.mmap.23
-rw-------  1 www-data www-data 1105926140 May  3 18:37 zm.mmap.24
-rw-------  1 www-data www-data   46085740 May  3 18:37 zm.mmap.4
-rw-------  1 www-data www-data          0 May  3 18:47 zm.mmap.5
-rw-------  1 www-data www-data   27653580 May  3 18:47 zm.mmap.6
-rw-------  1 www-data www-data          0 May  3 18:47 zm.mmap.8
-rw-------  1 www-data www-data  311045740 May  3 18:37 zm.mmap.9

[marcmerlin]

So, here is debug for one failing camera:

05/03/20 19:31:36.874025 zmc_m19[24845].DB1-zm_logger.cpp/247 [LogOpts: level=DB1 effective=DB1, screen=OFF, database=OFF, logfile=DB1->/tmp/zm/zm_debug.log.24845, syslog=INF]
05/03/20 19:31:36.874105 zmc_m19[24845].DB1-zm_utils.cpp/285 [Detected a x86\x86-64 processor with AVX]
05/03/20 19:31:36.874200 zmc_m19[24845].DB1-zm_monitor.cpp/2096 [Capture Delay!? 333.000]
05/03/20 19:31:36.874217 zmc_m19[24845].DB1-zm_monitor.cpp/2118 [Got 1 for v4l_captures_per_frame]
05/03/20 19:31:36.875030 zmc_m19[24845].DB1-zm_remote_camera.cpp/107 [1 addresses returned]
05/03/20 19:31:36.875125 zmc_m19[24845].DB1-zm_monitor.cpp/423 [monitor purpose=1]
05/03/20 19:31:36.875147 zmc_m19[24845].DB1-zm_monitor.cpp/432 [mem.size(4) SharedData=600 TriggerData=560 VideoStoreData=4116 timestamps=400 images=50x6220800 = 1335906627747840000 total=1335931280860119040]
05/03/20 19:31:36.875175 zmc_m19[24845].DB1-zm_storage.cpp/94 [No id passed to Storage constructor. Using default path /var/cache/zoneminder/events instead]
05/03/20 19:31:36.875192 zmc_m19[24845].DB1-zm_monitor.cpp/441 [Storage path: /var/cache/zoneminder/events]
05/03/20 19:31:36.875244 zmc_m19[24845].DB1-zm_monitor.cpp/591 [Unable to map file /dev/shm/zm.mmap.19 (311045740 bytes) to locked memory, trying unlocked]
05/03/20 19:31:36.875267 zmc_m19[24845].DB1-zm_monitor.cpp/594 [Mapped file /dev/shm/zm.mmap.19 (311045740 bytes) to unlocked memory]
05/03/20 19:31:37.137810 zmc_m19[24845].DB1-zm_monitor.cpp/507 [Monitor cam-frontdoor2 has function 5,
label format = '%N - %d/%m/%y %H:%M:%S', label X = 0, label Y = 0, label size = 1,
image buffer count = 50, warmup count = 25, pre-event count = 25, post-event count = 25, alarm frame count = 1,
fps report interval = 1000, ref blend percentage = 6, alarm ref blend percentage = 6, track motion = 0]
05/03/20 19:31:37.138011 zmc_m19[24845].DB1-zm_zone.cpp/884 [Got 1 zones for monitor cam-frontdoor2]
05/03/20 19:31:37.142843 zmc_m19[24845].DB1-zm_monitor.cpp/2391 [Loaded monitor 19(cam-frontdoor2), 1 zones]
05/03/20 19:31:37.142879 zmc_m19[24845].INF-zmc.cpp/223 [Starting Capture version 1.34.8]
05/03/20 19:31:37.198214 zmc_m19[24845].WAR-zm_remote_camera_http.cpp/1081 [Unable to capture image, retrying]
05/03/20 19:31:37.198306 zmc_m19[24845].DB1-zm_remote_camera_http.cpp/656 [Timeout waiting for HEADERCONT]
05/03/20 19:31:37.198331 zmc_m19[24845].DB1-zm_remote_camera_http.cpp/656 [Timeout waiting for HEADERCONT]
05/03/20 19:31:37.198353 zmc_m19[24845].DB1-zm_remote_camera_http.cpp/656 [Timeout waiting for HEADERCONT]

[marcmerlin]

So, am I out of luck? I spent quite a while gathering those errors, and I'm no closer to my system being fixed affter too many months of being broken.
I'm honestly about to give and replace everything with shinnobi :-/

[hster]

This is unfortunately the bad side of open source. I upgraded today from 1.32>1.34 and the upgrade appeared to have wiped out all previous events and would not record the photos.

I saw that the filters got all messed up with the predicates being wiped out so all events were deleted as now everything matched.

Then the events would not record the photo. After an hour of debugging, I reverted and saw the events directory was set as root and chowning to www-data fixed it. Since I have have a really old SBC odroid running 14.04, I can see this solution probably will come to an end.

[marcmerlin]

honestly, I had gotten less and less happy with ZM over time, but at least it worked-ish.
Such breakage on upgrades that then goes and stays unfixed, just didn't work.

I ended up upgrading to shinobi which someone went through the work of censoring so that it shows up as 'yahoo' instead of 'shi''no''bi' and while I'm not going to lie that it took a while to setup, it works much much better so far, and uses a fraction of the CPU that ZM did wiht my 15 cameras (actually ZM would just drop so many frames that I would sometimes lose minutes of critical capture on my cameras).

I should have switched a long time ago. Thanks to the folks who worked on ZM 10+ years ago, it was useful for a while, but it just didn't keep up with the times.

[apolonio]

I decided to upgrade recently as well, I remember have to run zmudate.pl to fix my database and also there were some settings that did not change right like my filters were wrong.

Then I recently patched again and something changed my auth relay method to hashed from none, I forced myself to figure it out so I am actually better for it.

But I totally feel your frustration.

[marcmerlin]

Thanks for the answer orianajonathan.
Note that I was mostly doing jpeg pulls, and for a reason, I had enough cameras that ZM could not keep up with 15 ffmpeg streams (and some older cameras were jpeg pull only).
This worked perfectly fine for about 10 years until ZM broke it when I upgraded.
That said given that it was broken for over 4 months and I got no solution and it was still broken after upgrading ZM again (and not just one camera, 6 different cameras were broken when they worked fine with the same config for years before that), I moved to shinob i . Now everything works fine and I don't have the load issues I had with ZM anymore.

[iconnor]

I know I'm not the best support person around here... can barely keep up both time wise and emotionally...

but I'm 100% sure that your problem was likely simple to fix.

You are the only person having this problem, and as I fix people's problems every single day.... I don't understand why your issue went unfixed for 4 months and now you have to proclaim so bitterly about how we suck and Shinobi is the only answer.

Are you in fact Moe?

Seriously... I don't even have the time to go back in forum history and figure out why this went wrong, but I'm sorely tempted.
DO you have any interest in figure it out? If not, the go away. You aren't running ZonemInder anymore and your statements about how ZM couldn't handle 15 streams aren't helping anyone.

We are happy for you that Shinobi works for you. We make a point of not bashing Shinobi. We are happy that multiple solutions exist. Bash ZM and the devs one more time and we will ban you.

Have some couth.

[marcmerlin]

I see this is a sensitive subject.
Feel free to look up my history on this forum if it makes you feel better and the different problems I encountered with mocord, dropped frames (likely because my 8 core system with SSD couldn't keep up with the load ZM needed, it was working better when I only had 5 cameras that were lower res).
ZM also grew other problems over time, like not deleting old events, despite a purge background event that's been there the entire time and worked fine when I ran it by hand.
Obviously this is where you would tell me that I should file bugs, and you'd be right. I did, they got no reply, outsdie of a bot trying to close said bugs every few months or so.

As for my problem that was likely easy to fix, maybe it was, I spent days on it, asked for help multiple times, nothing worked. It's very possible that if I had wiped everything, lost all my settings and started from scratch, it might have worked better, hard to say.

That said, I didn't say ZM sucked, or that any developer sucked, It's not like I paid for it or that I have a support contract.
If you feel however that it's not ok for me to say how disapointed I am that ZM failed for me in the end after my using it for almost 10 years and sinking so much time into it, well, I guess that's your opinion.
If it's not ok for me to say that they were thieves in my street 3 times and that out of 3 cameras and 3 times, I only got 2 somewhat usable frames out of 9 chances, and never got any license plates because of dropped frames/events, well, I'm not sure what to say, it still happened and my wife was pretty pissed.
With direct H264 recording passthrough to the same SSD on the same system, everything is now being recorded without any loss.