Hi Phil an all,
I have a system in service with 6 cameras, all in modect mode. Occasionally one or more cameras will stop recording events. Different camera(s) each time.This can happen after a day or two of uptime or a couple of weeks. Streaming from these cameras continues to work. If "force alarm" is selected, the "cancel forced alarm" link appears, but the monitor does not go into alarm status, and no event is generated. In this condition attemting to restart ZM hangs. No errors in syslog. "service zm restart" "zmpkg.pl restart" or from the web interface just hang, no error messages.
If the machine is rebooted, everything comes back up working properly. I have hacked the MySQL init script to check and repair all tables on startup. I suspect a database problem is the cause, or possibly a hung zma proccess? This problem has persisted through a couple of ZM upgrades, currently 1.19.3. Debugging this sytem is difficult as I have no remote access and working on it involves a 45 minute drive. The owner is generally anxious to get it back up quickly so he reboots it and it is working correctly when I am able to work on it.
Has anyone else seen this behavior? Suggestions for debugging?
Regards,
Ross
monitor stops recording events
-
zoneminder
- Site Admin
- Posts: 5215
- Joined: Wed Jul 09, 2003 2:07 pm
- Location: Bristol, UK
- Contact:
Hi Ross,
I've not seen this before. I think you are correct in that possibly it's worth checking zma as zmc will get restarted if it locks up. As a workaround it may be worth doing a 'zmdc.pl restart zma -m <id>' regularly in a cron job to see if it goes away, or even a 'zmpkg.pl restart'.
If you have access to the machine at any time while this is happening you could try sending a SIGABRT to the zma process to try and generate a core dump which _might_ have something useful in it.
Phil,
I've not seen this before. I think you are correct in that possibly it's worth checking zma as zmc will get restarted if it locks up. As a workaround it may be worth doing a 'zmdc.pl restart zma -m <id>' regularly in a cron job to see if it goes away, or even a 'zmpkg.pl restart'.
If you have access to the machine at any time while this is happening you could try sending a SIGABRT to the zma process to try and generate a core dump which _might_ have something useful in it.
Phil,
Phil,
Thanks for the suggestions. I've been experimenting to try and reproduce this problem on my home system.
If I send a zma process a SIGSTOP the web interface becomes useless. Even though other monitors will continue to capture events, nothing can be viewed in the browser. Any new window opened is blank. This is more extreme than the problem in my previous post, as that system mostly continues to function normally. Sending a SIGCONT to the zma process does not restore function to the web interface. As with the original problem, "zmpkg.pl restart" or "stop" just hangs. I have not found any way to restore function short of a system reboot.
It seems like there should be a way to restart zm unconditionally, regardless of a hung or stopped process.
Best regards,
Ross
Thanks for the suggestions. I've been experimenting to try and reproduce this problem on my home system.
If I send a zma process a SIGSTOP the web interface becomes useless. Even though other monitors will continue to capture events, nothing can be viewed in the browser. Any new window opened is blank. This is more extreme than the problem in my previous post, as that system mostly continues to function normally. Sending a SIGCONT to the zma process does not restore function to the web interface. As with the original problem, "zmpkg.pl restart" or "stop" just hangs. I have not found any way to restore function short of a system reboot.
It seems like there should be a way to restart zm unconditionally, regardless of a hung or stopped process.
Best regards,
Ross
-
zoneminder
- Site Admin
- Posts: 5215
- Joined: Wed Jul 09, 2003 2:07 pm
- Location: Bristol, UK
- Contact:
Hi Ross,
Hmm, I will have a look at this. zmpkg just uses zmdc which should send a SIGTERM to all it's processes and then a SIGKILL to any that don't obey. It shouldn't be too hard to shut down ZM and then just start one zma process (any maybe a zmc) and then stop it as you've described and see what the zmdc log says when trying to 'stop' it with that. This may shed some light on why it's not terminating, it may have something to do with non-reaped children I suspect.
Phil,
Hmm, I will have a look at this. zmpkg just uses zmdc which should send a SIGTERM to all it's processes and then a SIGKILL to any that don't obey. It shouldn't be too hard to shut down ZM and then just start one zma process (any maybe a zmc) and then stop it as you've described and see what the zmdc log says when trying to 'stop' it with that. This may shed some light on why it's not terminating, it may have something to do with non-reaped children I suspect.
Phil,