ZM captures gone crazy haywire - images from the past appearing

[scooty]

Hi all,

ZM has been up and running for a few days now. It's giving me some strange captures.

Looking at captures in the console, the dates displayed are wrong, the latest date shown is yesterday, for recordings taken today. I've checked the server system date, time zone and time, this is all correct. Looking closer at the displayed times for events, they are shown as last night, when in reality they are from this afternoon, ie, the displayed date and time of events I've just looked at is roughly 18 hours ago, for events just recorded!

Browsing the HDD, I can see that ZM keeps on making capture folders for new events, several folders every second, but and saving into them images from some point in the past! This is clearly obvious as the images are black&white from night time, and it's presently day time. It may be the images ZM is saving now, are form last night, ref 18 hour issue above)

ZM is creating LOT of folders, containing just two or three images, all the same, identical night time shot. It's saving between approx 4 and 50 event folders a second, then has a pause for a minute or so, and then repeats! But each folder contains images from the past!!!

Here is a list of some of the latest event folders ZM is creating, many per second... Note how Linux has given me correct time and date that the folder was created, but the times of the events in the console are all in the past. The folder name is yesterday (today is the 3rd): /cctv/Front%20Door/2024-09-02

Code: Select all

drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659720
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659721
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659722
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659723
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659724
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659725
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659726
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659727
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659728
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659729
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659730
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659731
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659732
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659733
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659734
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659735
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659736
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659737
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659738
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659739
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659740
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659741
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659742
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659743
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659744
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659745
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659746
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659747
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659748
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659749
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659750
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659751
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659752
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659753
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659754
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659755
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659756
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659757
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659758
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659759
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659760
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659761
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659762
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659763
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659764
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659765
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659766
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659767
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659768
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659769
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659770
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659771
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659772
drwxr-xr-x 1 scott scott 4096 Sep  3 15:46 1659773
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659774
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659775
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659776
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659777
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659778
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659779
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659780
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659781
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659782
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659783
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659784
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659785
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659786
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659787
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659788
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659789
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659790
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659791
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659792
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659793
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659794
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659795
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659796
drwxr-xr-x 1 scott scott 4096 Sep  3 15:47 1659797
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659800
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659802
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659804
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659806
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659808
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659809
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659810
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659811
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659812
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659813
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659814
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659815
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659816
drwxr-xr-x 1 scott scott 4096 Sep  3 15:48 1659817
drwxr-xr-x 1 scott scott 4096 Sep  3 15:57 1659842
drwxr-xr-x 1 scott scott 4096 Sep  3 15:57 1659844
drwxr-xr-x 1 scott scott 4096 Sep  3 15:57 1659845
drwxr-xr-x 1 scott scott 4096 Sep  3 15:57 1659846

These events roughly correlate to these events in the console, not the dates and times and black and white thumbnails from last night:

Screenshot 2024-09-03 at 16-08-56 ZM - Events.png (563.35 KiB) Viewed 2816 times

ZM resource use in htop when saving all these folders:

Screenshot from 2024-09-03 16-41-23.png (224.35 KiB) Viewed 2805 times

Is there a simple way to just clear the ZM capture database, and delete all the capture folders and images, and allow ZM to start capturing afresh?

Cheers, Scott.

[scooty]

I've tried to look at the ZM Log, but just get:

Loading, please wait...

[scooty]

I just did

sudo service zoneminder stop

and the load really dropped, compare htop to the previous screenshot:

Screenshot from 2024-09-03 17-16-02.png (212.61 KiB) Viewed 2796 times

Will keep on poking around until any good ideas come up.

I want to get this working well ASAP base on our neighbour being robbed the other day.

Thanks in advance...

[scooty]

Today's update...

Now when from the console, I click into hourly/daily/weekly captures, I get this error, or slight variations of it:

zm-error1.jpg (26.97 KiB) Viewed 2755 times

As text:

SQL-ERR '00000

', statement was 'SELECT E.*, M.Name AS Monitor FROM `Events` AS E INNER JOIN Monitors AS M ON E.MonitorId = M.Id WHERE ( E.StartDateTime >= '2024-09-03 13:41:04'
and E.MonitorId = '1'
) ORDER BY E.Id DESC'

[scooty]

Can anyone suggest a way to reset the captures database and delete the old images?

Or any other ideas please?

[dougmccrary]

not sure what

reset the captures database

means, but you could try running zmaudit.pl to get rid of the junk from earlier.
It's not super fast, but it works.

[scooty]

Ah thanks for that, it gave me a clue:

Code: Select all

$ sudo zmaudit.pl
09/06/24 11:15:47.552534 zmaudit[646954].INF [main:199] [Auditing All Storage Areas]
DBD::mysql::st execute failed: The table 'Logs' is full at /usr/share/perl5/ZoneMinder/Logger.pm line 627.
09/06/24 11:15:47.553761 zmaudit[646954].ERR [ZoneMinder::Logger:630] [Can't execute log entry 'INSERT INTO Logs ( TimeKey, Component, ServerId, Pid, Level, Code, Message, File, Line ) VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, NULL )': The table 'Logs' is full]

[scooty]

Thanks, I ran the audit.

In the results there were quite a few lines like these, path not found, and no end time:

Code: Select all

7 Notes => Motion: All Orientation => ROTATE_270 SaveJPEGs => 3 Scheme => Medium StartDateTime => 2024-09-04 23:06:01 StateId => 1 StorageId => 2 Width => 1080]
09/06/24 11:35:59.802367 zmaudit[646954].WAR [main:632] [ZoneMinder::Event: Id => 1677917 MonitorId => 6 Scheme => Medium StartDateTime => 2024-09-05 15:21:48 StorageId => 2]
09/06/24 11:35:59.827012 zmaudit[646954].WAR [ZoneMinder::Event:550] [Path not found for Event 1664222 at /mnt/cctv/6/2024-09-03/1664222]
09/06/24 11:35:59.827183 zmaudit[646954].WAR [main:630] [Not found at /mnt/cctv/6/2024-09-03/1664222 was found at /mnt/cctv/6/2024-09-04/1664222]

Code: Select all

09/06/24 11:36:00.631612 zmaudit[646954].WAR [main:632] [ZoneMinder::Event: Id => 1663282 MonitorId => 6 Scheme => Medium StartDateTime => 2024-09-04 02:23:55 StorageId => 2]
Event 1559623 has no end time and is 559630 seconds old. Deleting it., deleting

To boot, unrelated to running the audit as this happened just prior, I now have no web interface, just a white screen. I tried stopping and starting the service, it didn't help to recover the web interface.

Now when I look at the status of zm service at the command line I get:

Code: Select all

$ sudo service zoneminder status
● zoneminder.service - ZoneMinder CCTV recording and surveillance system
     Loaded: loaded (/lib/systemd/system/zoneminder.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-09-06 11:14:00 BST; 50min ago
    Process: 646820 ExecStart=/usr/bin/zmpkg.pl start (code=exited, status=0/SUCCESS)
   Main PID: 646834 (zmdc.pl)
      Tasks: 18 (limit: 38415)
     Memory: 2.9G
        CPU: 1h 42min 23.573s
     CGroup: /system.slice/zoneminder.service
             ├─646834 /usr/bin/perl -wT /usr/bin/zmdc.pl startup
             ├─646862 /usr/bin/zmc -m 1
             ├─646867 /usr/bin/zmc -m 4
             ├─646877 /usr/bin/zmc -m 6
             ├─646882 /usr/bin/perl -wT /usr/bin/zmfilter.pl --filter_id=1 --daemon
             ├─646887 /usr/bin/perl -wT /usr/bin/zmfilter.pl --filter_id=2 --daemon
             ├─646892 /usr/bin/perl -wT /usr/bin/zmwatch.pl
             ├─646896 /usr/bin/perl -wT /usr/bin/zmupdate.pl -c
             └─646901 /usr/bin/perl -wT /usr/bin/zmstats.pl

Sep 06 12:04:56 hal zmc_m6[646877]: ERR [zmc_m6] [Can't run query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1725620696.644157, 'zmc_m6', 0, 646877, 0, 'INF', 'Ca>
Sep 06 12:04:56 hal zmc_m5[649862]: ERR [zmc_m5] [Invalid font location: /usr/share/zoneminder/www/fonts/default.zmfnt]
Sep 06 12:04:56 hal zmc_m5[649862]: ERR [zmc_m5] [Can't run query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1725620696.653355, 'zmc_m5', 0, 649862, -2, 'ERR', 'I>
Sep 06 12:04:56 hal zmc_m5[649862]: INF [zmc_m5] [Starting Capture version 1.36.34]
Sep 06 12:04:56 hal zmc_m5[649862]: ERR [zmc_m5] [Can't run query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1725620696.660096, 'zmc_m5', 0, 649862, 0, 'INF', 'St>
Sep 06 12:04:56 hal zmc_m4[646867]: INF [zmc_m4] [Front Door: 48900 - Capturing at 16.06 fps, capturing bandwidth 431941bytes/sec Analysing at 16.06 fps]
Sep 06 12:04:56 hal zmc_m4[646867]: ERR [zmc_m4] [Can't run query INSERT INTO `Logs` ( `TimeKey`, `Component`, `ServerId`, `Pid`, `Level`, `Code`, `Message`, `File`, `Line` ) VALUES ( 1725620696.711135, 'zmc_m4', 0, 646867, 0, 'INF', 'Fr>
Sep 06 12:04:56 hal zmdc[646834]: INF ['zmc -m 5' sending stop to pid 649862 at 24/09/06 12:04:56]
Sep 06 12:04:56 hal zmc_m5[649862]: ERR [zmc_m5] [Unable to open input rtsp://192.168.1.55:554/cam/realmonitor?Channel=1&subtype=0?tcp due to: Immediate exit requested]
Sep 06 12:04:56 hal zmdc[646834]: INF ['zmc -m 5' exited normally]

Is there any way to recover this, or is it delete and fresh install time?

[scooty]

Here are some capture examples,

Type 1
please look at the images in the capture - one from daytime, 14:00, then a few from midnight.

Screenshot 2024-09-14 at 00-13-45 ZM - Frames - 1787227.png (144.08 KiB) Viewed 2443 times

The time delta is interesting, as it shows ~10 hours between the daylight image and the row of night time images.

Type 2

I have a shed load of captures that are just one frame long, this is one example:

Screenshot 2024-09-14 at 00-19-20 ZM - Frames - 1787182.png (33.01 KiB) Viewed 2443 times

Looking at the captures, ZM saves a row of separate captures (like above) just milliseconds after each other, instead of them all being part of one capture lasting a few seconds.

These two scenarios are happening a lot. Does anyone have any ideas as to why?

Thanks, Scott

[iconnor]

ZM always has to start a recording with a keyframe. So the queue ALWAYS keeps a keyframe at the head. If a system is overloaded, we drop packets, but we drop them after that first keyframe.

So you can get into a situation where there is a large time discrepancy between the first frame of a recording and the second. Normally that would be like a few minutes though. Yours is hours out.



Your system is totally screwed. I can only guess that it is hopelessly underpowered.

[MechanicMike]

Yo Scott! Sounds like your ZM captures are on a wild ride. To clear the capture database and start fresh, you can try accessing the database directly and deleting the old entries. Make sure to back up everything first in case things go south. Also, check your ZM settings to ensure the time settings are accurate to prevent this funky time-traveling capture issue in the future. Hope this helps, bro!

[scooty]

Thanks Both... reading around, I found a thread stating that there are three clocks that need to be in sync.. the main system clock, a PHP clock and SQL clock, so in order to check them I started with the PHP clock.

(thread: viewtopic.php?p=130331#p130331)

I cobbled together a quick PHP script to print the PHP time and date to the terminal, then printed the system time and date manually within a second:

Code: Select all

$ php php-date-test.php 
 PHP Date and Time: Monday - 2024-09-16 - 03:47:43pm - Timezone Europe/London 
$ date
Mon Sep 16 15:47:43 BST 2024

For reference, this was the simple script (named: php-date-test.php) to test the PHP time and date and timezone:

Code: Select all

 <?php
echo "PHP Date and Time: " . date("l") . " - " ;
echo date("Y-m-d") . " - ";
echo date("h:i:sa") . " - " ;
echo "Timezone " . date_default_timezone_get();
?> 

Now I have to find a way to check the SQL time aligns too....

Cheers, Scott

[iconnor]

ZM will be logging errors if there is a mismatch. Check your logs.