Create video for all matches - Filters

[burger]

tl;dr Create video for all matches is legacy code, and won't make sense with passthrough. There are some checks for whether the video was already made that need to be removed. And then ideally the code would concatenate all the videos into one. Maybe the option should just be removed since it doesn't work with modern cameras (non jpeg), and will likely frustrate users. They are better off just making a bash script for this (which can be as simple as just looking up all mp4 events in a modect camera folder within a given time range, and then concatenating them via ffmpeg in cron). That's what I will do, instead of trying to wrangle with the db sql queries, and the perl scripts. Although this loses some functionality (ability to filter based on detections for cameras that have lots of events).

Here's the full log:

Code: Select all

zmfilter_35[6833].DB1 [ZoneMinder::Filter:101] [Filter::Execute SQL (SELECT E.*, unix_timestamp(E.StartDateTime) as Time, M.Name as MonitorName
          FROM Events as E INNER JOIN Monitors as M on M.Id = E.MonitorId WHERE (  M.Name = '40_Hotroom_Hallway'
 and E.EndDateTime >= '2024-10-28 22:57:26'
 ) AND ( E.Videoed = 0 ) ORDER BY E.StartDateTime ASC LIMIT 0,50)]
10/29/24 22:57:26.738290 zmfilter_35[6833].DB1 [ZoneMinder::Filter:114] [Loaded 0 events for filter Create Videos for Last Day of Hotroom Hallway using query (SELECT E.*, unix_timestamp(E.StartDateTime) as Time, M.Name as MonitorName
          FROM Events as E INNER JOIN Monitors as M on M.Id = E.MonitorId WHERE (  M.Name = '40_Hotroom_Hallway'
 and E.EndDateTime >= '2024-10-28 22:57:26'
 ) AND ( E.Videoed = 0 ) ORDER BY E.StartDateTime ASC LIMIT 0,50)"]
10/29/24 22:57:26.738409 zmfilter_35[6833].DB1 [main:303] [Checking filter Create Videos for Last Day of Hotroom Hallway video returned 0 events ]

This outputs 0 events. debug on the filter webpage says:

Code: Select all

SQL SELECT E.*,M.Name AS MonitorName,M.DefaultScale
FROM Monitors AS M INNER JOIN Events AS E ON (M.Id = E.MonitorId)
WHERE
M.Name = '40_Hotroom_Hallway' and E.EndDateTime >= '2024-10-28 22:58:13' ORDER BY StartDateTime ASC LIMIT 50

This outputs the correct 15 or so events.

On a whim I looked up the videoed flag, as it looks suspicious.

I think the problem is in zmfilter.pl

Code: Select all

  if ( $Config{ZM_OPT_FFMPEG} && $filter->{AutoVideo} ) {
      if ( !$Event->{Videoed} ) {
        $delete_ok = undef if !generateVideo($filter, $Event);

So I changed it to:

Code: Select all

  if ( $Config{ZM_OPT_FFMPEG} && $filter->{AutoVideo} ) {
      if ( !$Event->{Archived} ) {
        $delete_ok = undef if !generateVideo($filter, $Event);
      }

That didn't work because the Videoed is taken from somewhere else.

I found it by looking at the Debian package.
/usr/share/perl5/ZoneMinder
and commented out:

Code: Select all

#    if ( $self->{AutoVideo} ) {
#      push @auto_terms, 'E.Videoed = 0';
#    }

I didn't even know these perl scripts were there. Or if I did, I never really looked at them. And now the filter finds the events.

Code: Select all

10/29/24 23:40:46.623078 zmfilter_35[16986].DB1 [ZoneMinder::Logger:321] [LogOpts: level=DB9/DB9, screen=OFF, database=OFF, logfile=DB9->/var/log/zm/zmfilter_debug.16986, syslog=OFF]
10/29/24 23:40:46.688590 zmfilter_35[16986].INF [main:184] [Scanning for events using filter id '35']
10/29/24 23:40:46.688677 zmfilter_35[16986].DB1 [main:205] [Reloading filters]
10/29/24 23:40:46.689311 zmfilter_35[16986].DB1 [main:264] [Found filter 'Create Videos for Last Day of Hotroom Hallway']
10/29/24 23:40:46.737485 zmfilter_35[16986].DB1 [main:278] [Got 1 filters]
10/29/24 23:40:46.738854 zmfilter_35[16986].DB1 [ZoneMinder::Filter:101] [Filter::Execute SQL (SELECT E.*, unix_timestamp(E.StartDateTime) as Time, M.Name as MonitorName
          FROM Events as E INNER JOIN Monitors as M on M.Id = E.MonitorId WHERE (  M.Name = '40_Hotroom_Hallway'
 and E.EndDateTime >= '2024-10-28 23:40:46'
 ) ORDER BY E.StartDateTime ASC LIMIT 0,50)]
10/29/24 23:40:47.302751 zmfilter_35[16986].DB1 [ZoneMinder::Filter:114] [Loaded 16 events for filter Create Videos for Last Day of Hotroom Hallway using query (SELECT E.*, unix_timestamp(E.StartDateTime) as Time, M.Name as MonitorName
          FROM Events as E INNER JOIN Monitors as M on M.Id = E.MonitorId WHERE (  M.Name = '40_Hotroom_Hallway'
 and E.EndDateTime >= '2024-10-28 23:40:46'
 ) ORDER BY E.StartDateTime ASC LIMIT 0,50)"]
10/29/24 23:40:47.302861 zmfilter_35[16986].DB1 [main:303] [Checking filter Create Videos for Last Day of Hotroom Hallway video returned 16 events ]
10/29/24 23:40:47.302945 zmfilter_35[16986].DB1 [main:313] [Checking event 4089400]
10/29/24 23:40:47.303118 zmfilter_35[16986].DB1 [main:313] [Checking event 4089427]
10/29/24 23:40:47.303278 zmfilter_35[16986].DB1 [main:313] [Checking event 4091260]
10/29/24 23:40:47.303436 zmfilter_35[16986].DB1 [main:313] [Checking event 4091271]
10/29/24 23:40:47.303595 zmfilter_35[16986].DB1 [main:313] [Checking event 4091500]
10/29/24 23:40:47.303761 zmfilter_35[16986].DB1 [main:313] [Checking event 4091501]
10/29/24 23:40:47.303843 zmfilter_35[16986].DB1 [main:313] [Checking event 4092738]
10/29/24 23:40:47.303919 zmfilter_35[16986].DB1 [main:313] [Checking event 4092783]
10/29/24 23:40:47.303997 zmfilter_35[16986].DB1 [main:313] [Checking event 4092809]
10/29/24 23:40:47.304074 zmfilter_35[16986].DB1 [main:313] [Checking event 4092852]
10/29/24 23:40:47.304150 zmfilter_35[16986].DB1 [main:313] [Checking event 4092904]
10/29/24 23:40:47.304226 zmfilter_35[16986].DB1 [main:313] [Checking event 4094195]
10/29/24 23:40:47.304303 zmfilter_35[16986].DB1 [main:313] [Checking event 4094652]
10/29/24 23:40:47.304380 zmfilter_35[16986].DB1 [main:313] [Checking event 4094668]
10/29/24 23:40:47.304455 zmfilter_35[16986].DB1 [main:313] [Checking event 4094695]
10/29/24 23:40:47.304529 zmfilter_35[16986].DB1 [main:313] [Checking event 4094832]

But it stops there.

This is still in master, so it probably should be fixed, given that no one uses savejpeg anymore
and its all saved as h264 which sets the Videoed flag to 1, I am assuming.

And it looks like main here refers to zmfilter.pl, and I guess I still need to change that
Videoed lookup. Which I did by changing it to Archived instead of Videoed. And it now makes the videos.

Code: Select all

10/29/24 23:50:50.202403 zmfilter_35[18762].DB1 [ZoneMinder::Object:126] [Loading ZoneMinder::Monitor from Monitors WHERE Id = 40]
10/29/24 23:50:52.249106 zmfilter_35[18762].DB1 [main:460] [Output: /videos/Four_6tb1/40/2024-10-29/4089400/Event-4089400-r1-s1.mp4]
10/29/24 23:50:52.250162 zmfilter_35[18762].DB1 [main:313] [Checking event 4089427]
10/29/24 23:50:53.913155 zmfilter_35[18762].DB1 [main:460] [Output: /videos/Four_6tb1/40/2024-10-29/4089427/Event-4089427-r1-s1.mp4]
10/29/24 23:50:53.914113 zmfilter_35[18762].DB1 [main:313] [Checking event 4091260]
10/29/24 23:50:55.566843 zmfilter_35[18762].DB1 [main:460] [Output: /videos/Four_6tb1/40/2024-10-29/4091260/Event-4091260-r1-s1.mp4]

I checked the video and it plays. However, I have passthrough set, with the resolution at
320x240, so the output is smaller than the passthrough video which is 2K or so. I suppose
that's the least of the problems here.

Users would probably want to get one single video made of all events. That was my intention.
I guess I'll just have to make a manual bash or python script to do this.

So there is some legacy code here that should at least, get the videoed lookups removed.
Maybe the option should just be removed, to save people the headache of troubleshooting
why it doesn't work. In any case, that is the current state of Create video for all matches.

[iconnor]

Problem is, there are people still saving jpegs. Some are still running 1.24. Scary stuff.

I do agree that we can remove this though. Functionality should be provided by external script.

[burger]

After thinking about it a bit, I came to the following conclusions.

1) Create video for all matches, can mean different things to different
people. In my case, I intended to create a single video that concatenated
a number of events into one video, so that I could watch the single video
from my desktop, as opposed to the GUI. I find this a better viewing experience.

Though I understand the original developers idea to "Create Videos for all matches"
where they intended to encode JPGs into an MP4. That is a different use case.

2) I think it's OK to leave the current functionality, but it should be renamed to
remove any misconception about what the command does. All that is needed is
to say something like "Create MP4 videos for JPG events" on the Filter page. Note
that it's not 'video', but 'videos', which is a subtle but important difference. If
you say that people are running old ZM, or have old cameras, then that is more reason
to leave some backwards compatibility in. Backwards compatibility is good (as long
as it's not a maintenance burden).

3) I said it was easy enough to bash some solution in for my use case, so here it is
after 30-45 minutes of web searching and copy and paste hacking. What I do is take
the videos and send them to a separate server to do the ffmpeg concatenation. Then I can
download them each day and review the videos. (I may end up using a VPS, as this
taxes the CPU quite a bit/uses more power, and they are better equipped for this.)

Code: Select all

ssh user@host rm /tmp/list.txt
ssh user@host rm -rf /tmp/videos_of_camera
ssh user@host mkdir /tmp/concat_videos/
ssh user@host mkdir /tmp/videos_of_camera/
find /videos/hdd/monitornumber -type f -name "*.mp4"  -mtime -1 -exec scp -pr {} user@host:/tmp/videos_of_camera/. \;
ssh user@host touch /tmp/list.txt
ssh user@host 'for f in /tmp/videos_of_camera/*.mp4 ; do echo file $f >> /tmp/list.txt; done'
ssh user@host "ffmpeg -y -f concat -safe 0 -i /tmp/list.txt -movflags faststart /tmp/concat_videos/video.mp4"

Ideally, this script would be run through the filters web interface, but it's also OK
to just hack something together, if it will work given the constraints. I only needed
the last 24 hours of modect events, so a simple find command does the job. Run this in
a crontab, and you can get the videos every day.

It shouldn't need to be said, but a script like the above only makes sense with modect events.
Otherwise, you will have 24 hours of video.

And I find this a very powerful tool to watch footage. Played back at something like 5x speed on mpv, it's a killer feature for watching a large number of events. Much better and practical than the web interface through gapless mode (web browsers have always been slow).

EDIT: here is the current script I'm using with loops to make it easier to manage multiple cameras.

Code: Select all

logger "video create, folder maintenance"
ssh user@ipaddress rm -rf /tmp/concat_videos/
ssh user@ipaddress mkdir /tmp/concat_videos/
for MONITORNUMBER in 13 34 37 38 40 46 47 49 ;
do
  echo "creating folders for $MONITORNUMBER"
  ssh user@ipaddress rm      /tmp/list_$MONITORNUMBER.txt
  ssh user@ipaddress rm -rf  /tmp/videos_$MONITORNUMBER/
  ssh user@ipaddress mkdir   /tmp/videos_$MONITORNUMBER/
  ssh user@ipaddress touch   /tmp/list_$MONITORNUMBER.txt
done
logger "video create, send movies"
echo "sending movies"
#hard coded, because we don't know what storage area they will be on
#technically you can do a glob here with a loop, but i just left it this way
find /videos/hdd1/13/    -type f -name "*.mp4"  -mtime -1 -exec scp -pr {} user@ipaddress:/tmp/videos_13/. \;
find /videos/hdd2/34/    -type f -name "*.mp4"  -mtime -1 -exec scp -pr {} user@ipaddress:/tmp/videos_34/. \;
find /videos/hdd1/37/    -type f -name "*.mp4"  -mtime -1 -exec scp -pr {} user@ipaddress:/tmp/videos_37/. \;
find /videos/hdd1/38/    -type f -name "*.mp4"  -mtime -1 -exec scp -pr {} user@ipaddress:/tmp/videos_38/. \;
find /videos/hdd3/40/    -type f -name "*.mp4"  -mtime -1 -exec scp -pr {} user@ipaddress:/tmp/videos_40/. \;
find /videos/hdd3/46/    -type f -name "*.mp4"  -mtime -1 -exec scp -pr {} user@ipaddress:/tmp/videos_46/. \;
find /videos/hdd3/47/    -type f -name "*.mp4"  -mtime -1 -exec scp -pr {} user@ipaddress:/tmp/videos_47/. \;
find /videos/hdd2/49/    -type f -name "*.mp4"  -mtime -1 -exec scp -pr {} user@ipaddress:/tmp/videos_49/. \;
for MONITORNUMBER in 13 34 37 38 40 46 47 49;
do
  ssh user@ipaddress 'for f in /tmp/videos_'$MONITORNUMBER'/*.mp4 ; do echo file $f >> /tmp/list_'$MONITORNUMBER'.txt; done'
done
logger "video create, making videos"
#2nd half of script will make the videos
#the first half of the script is just to copy all the files in one operation to get the timestamps close enough together

for monitornumber in 40 46 47 34 13 37 38 49;
do
  echo   "creating video for monitor $monitornumber"
  logger "video create, creating video for monitor $monitornumber"
  ssh user@ipaddress "ffmpeg -y -f concat -safe 0 -i /tmp/list_$monitornumber.txt -movflags faststart /tmp/concat_videos/my_cameras_$monitornumber.mp4"
done

logger  "video create, everything done. sending confirmation email."
export  EMAIL=address@website.com
echo    "videos created for watched cameras: scp -r /tmp/videos* ." | msmtp -a default destination@address.com