Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach when restarting ZM

[buedi]

Hello everyone,

I am running ZM (currently 1.36.25) for a few weeks now on a Ubuntu Server 20.04.5 LTS without any issues. I am still learning stuff, but so far it runs stable and yesterday I decided to dip into zmeventnotification.
So I installed the latest stable (6.1.28) as described in the Documentation and followed precisely all the prerequisites (Python version, Perl Modules etc.). I "think" I got it running, but since i had problems receiving Push notifications I enabled debugging logs to dig a bit deeper of what is going on. I found an issue I will probably post in another thread after doing more research. But while debugging, changing config files and restarting zonememinder I ran into the issue here.

This problem only started after installing and enabling zmeventnotification yesterday.
When the ZM machine is freshly booted ZM runs fine and I can do a 'systemctl restart zoneminder' as often as my heart desires without any problems.
But as soon as a single alarm triggered and I try a 'systemctl restart zoneminder' again, the command never finishes and I see those in the syslog:

Code: Select all

systemctl restart zoneminder

Sep 10 07:58:02 flathead zmeventnotification[31613]: ERR [Shared data not valid for monitor 1]
Sep 10 07:58:02 flathead zmeventnotification[31613]: ERR [|----> FORK:Logitech C310 (1), eid:836 SHM failed, re-validating it]
Sep 10 07:58:02 flathead zmeventnotification[31613]: ERR [Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach.  zmc might not be running.]
Sep 10 07:58:03 flathead zmeventnotification[31613]: ERR [Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach.  zmc might not be running.]
Sep 10 07:58:03 flathead zmeventnotification[31613]: ERR [|----> FORK:Logitech C310 (1), eid:836 SHM failed, re-validating it]
Sep 10 07:58:03 flathead zmeventnotification[31613]: ERR [Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach.  zmc might not be running.]
Sep 10 07:58:05 flathead zmeventnotification[31613]: ERR [Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach.  zmc might not be running.]
Sep 10 07:58:05 flathead zmeventnotification[31613]: ERR [|----> FORK:Logitech C310 (1), eid:836 SHM failed, re-validating it]
Sep 10 07:58:05 flathead zmeventnotification[31613]: ERR [Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach.  zmc might not be running.]
Sep 10 07:58:07 flathead zmeventnotification[31613]: ERR [Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach.  zmc might not be running.]
Sep 10 07:58:07 flathead zmeventnotification[31613]: ERR [|----> FORK:Logitech C310 (1), eid:836 SHM failed, re-validating it]
Sep 10 07:58:07 flathead zmeventnotification[31613]: ERR [Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach.  zmc might not be running.]
Sep 10 07:58:09 flathead zmeventnotification[31613]: ERR [Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach.  zmc might not be running.]
Sep 10 07:58:09 flathead zmeventnotification[31613]: ERR [|----> FORK:Logitech C310 (1), eid:836 SHM failed, re-validating it]
Sep 10 07:58:10 flathead zmeventnotification[31613]: ERR [Memory map file '/dev/shm/zm.mmap.1' does not exist in zmMemAttach.  zmc might not be running.]

To make matters worse, I can not even reboot the Machine via SSH. I have to do a hard power cycle to get it back up running.

I checked /dev/shm/zm.mmap.1 and zm.mmap.2 and they are owned by www-data:www-data and set to 660.

I assume I might have had that problem since I use ZM, but it never got triggered because /dev/shm is only used since yesterday when I started using zmeventnotification, right?

My question: What could be my problem here?

Thank you very much in advance.

Edit: The error in syslog only shows up as soon as I try the systemctl restart zoneminder. While triggering Alarms, they do not show up yet (although /dev/shm is used to store data there, right?). So I should not have a permission issue with /dev/shm?

[Magic919]

Turn off ZMES in ZM. See if that’s where the problem lies.

[buedi]

Yes, there is 0 problems with ES off in the System settings. No matter how many alarms are triggered, a systemctl restart zoneminder works without issue. As soon as ES is on and at least one alarm triggered, that is where the issue with /dev/shm starts

[Magic919]

Run through the manual triggering and troubleshooting of ZMES. Something will show up.

[buedi]

I made adjustments to the zmeventnotification.ini to have a cleaner debug log.
I disabled Hooks for now, until i have time to look into them and because of the current situation also disabled FCM_v1, so the "classic" Push notification is used instead.

I used

Code: Select all

sudo -u www-data /usr/bin/zmeventnotification.pl --debug

to start Debug and triggered an alert, which got me a Push notification and as far as I was able to see, no errors in the Debug output. I tried it a few times and it looked good. Finally did a systemctl restart zoneminder and that worked without an issue too.

So I went back to daemon mode for ES (debugging to /var/log/zm/zmeventnotification.log enabled) and triggered another Alert. Again not error I could spot and I received a Push notification. Finally I did a systemctl restart zoneminder and the error about /dev/shm/zm.map.1 was back again and I had to hard reset the Machine, because a clean reboot was also not possible.

Until now I was pretty sure that everything runs under www-data and everything I did so far (for example giving www-data access to the needed certificates) look like it does run under www-data. But something is off when ES runs as daemon, even with no visible error message in the Debug log. I set the Debug to file to Level 4 in the ZM System settings. Would it help to bump that up to 11 (9?). If I manually start ES with --debug, is that also Level 9?

[Magic919]

I think this bit covers all Asker had to say on troubleshooting ZMES https://zmeventnotification.readthedocs ... g-problems

Sounds like you are close, but it’s an odd problem. I’d also expect it to all run as www-data.

[buedi]

Thanks Magic919! I was away, but will dig into the logs deeper now. I will rise the Debug Level to 9 and see if this yields more information.
I picked the Picture URL from the Log this morning and tried to access it via the browser (putting in the Password in the URL of course), but that got me a 404. As far as I understood, the whole process uses the SHM device to exchange data between ZM and ES. If ZM was not able to put the picture into SHM, ES can not pick it up and I will also not be able to get it via URL. Maybe this is where the whole things collapses, not being able to write to /dev/shm for some odd reason.

As said, I will rise the Debug level and go through the various logs. Maybe it is one of the ZM processes locking up while trying to put data into /dev/shm and I only discovered it now after a few weeks, because /dev/shm was never used until I started using ES.

I will post my findings, hopefully with a solution

[buedi]

OK, so this is getting really, really weird now. I spent the last few hours trying to figure out what the issue is with the /dev/shm error and that Push Notifications are sometimes sent out and sometimes not and that no matter what I can not get it to include a picture (even static URL). This meant I did numerous restarts of ZM and config tweaks and I am not able to replicate the /dev/shm error message since about 2 hours. I just can trigger alarms and restart zm as often as I want, it just works.

In total, there is no difference at the moment configuration wise to what I had this morning. I remember this morning when I had the issue, I always was able to see a zmeventnotification.pl process which I also was not able to kill manually (I thought this one is blocking access to /dev/shm somehow), so I had to reboot / power cycle the PC where ZM runs on. But I do not even can see that anymore.

I am sorry that I can not tell why the problem stopped out of the blue. It seems like one of those issues you have and spend time to look very close and it just works again. Like it was just there to waste your time for nothing

[Magic919]

That is odd. Hopefully it’ll stay away.

In terms of zmeventnotification.pl not wanting to cease, I’ve experienced it, but it did respond to a kill -9.

[buedi]

Yeah, strange. I also tried with kill -9, no chance.
I saw that multiple instances of zmeventnotification.pl can run (but they seem to end after a while, except one that is always running), but I suppose this is normal? Maybe one per Alert and then they go away after a while?

[Magic919]

If I watch the process and PID, there is a running one and then an additional one spawns when I get an event and ceases afterwards. Whether that long-running one will refresh after a while, I don't know.

[buedi]

Thank you! Then it seems to be alright on my end