Record High-res H264 streams 24/7 with low CPU Load

[russell_i_brown]

Even in RECORD mode with H264 passthrough enabled, Zoneminder extracts frames from the stream (confirmed by iconner in this thread). This obviously creates a system load and a significant one when there's a number of high-res cameras involved.

The attached script uses ffmpeg to record the high-res H264 stream from cameras in N second chunks and inserts event records into the Zoneminder database for every chunk so that they can be accessed and managed via the normal interface.

For my setup, I'm recording high-res streams from six IP cameras 24/7 in ten minute chunks and doing MODECT on the low-res streams. This gives me motion events on the low-res and I can go and look at the 3840x2160 mp4 for the appropriate time if needed. This does mean you end with two cameras (low-res MODECT and high-res NONE) in the zoneminder console but hey-ho, I can live with that.

To set this up, firstly get a Monitor for your camera working in MONITOR mode at high-res (the script extracts the Height, Width, Path and StoragePath for a given Monitor from the DB).

Once working, set the new Monitor to 'NONE' so Zoneminder isn't running zmc on it.

Install the attached script (zmrecord.sh) in /usr/local/bin with execute perms.

Test it by running "zmrecord.sh <Monitor_Id>" and you should see it recording into <your_monitors_storage_path>/Monitor_Id/Rec-XXXXX. As each chunk is finished (set the chunk size in seconds by changing RECORD_TIME at the top of the script) , the script moves the mp4 to the right place in your storage, tells Zoneminder's DB all about it and starts recording the next chunk.

Copy the attached systemd (Ugh!) zmrecord@.service script to /lib/systemd/system (or wherever your systemd scripts live) and enable it for each camera with

Code: Select all

systemctl enable zmrecord@<MonitorId>.service

For multiple cameras (for example 8, 12 and 14) you would do:

Code: Select all

systemctl enable zmrecord@8.service zmrecord@12.service zmrecord@14.service

This will tell your system to fire up the ffmpeg recording once it's booted to multi-user mode. You can, of course, start or stop the zmrecord process for individual cameras through the normal systemctl commands:

Code: Select all

systemctl start zmrecord@12.service
      or
systemctl stop zmrecord@12.service

Here's a 'top' on my system recording high-res streams from six cameras with zmc/zma running on various low-res streams (the highest load are some old Axis cameras pulling mjpgs). The ffmpegs reading the high-res hardly feature whereas before, trying to record these high-res streams through Zoneminder, my system was on it's knees.

top.png (42.17 KiB) Viewed 484511 times

and here's a region of my Zoneminder console showing the 'None' monitors gathering events.

zm.png (58.18 KiB) Viewed 484511 times

Enjoy. Comments & suggestions welcome.


Note: the script will only work with Storage scheme 'Medium' (it's what I use).
Note2: You'll need to install the 'mediainfo' package as the script uses it to extract the frame count from the mp4.

[russell_i_brown]

Small update to the script - I screwed up setting the StorageId in the event. This will only effect you if you're using multiple storage areas.

Attached is version 0.3

There's also an issue with zmaudit getting enthusiastic about clearing out the events as there are no associated Frame records (Frames don't exist as we're recording native H264).

I need to do a little thinking about the right solution so for now I've just set AUDIT_MIN_AGE to 604800 (a week) while I come up with a sensible solution.

[Eiserfeyer]

Thanks for your work. Your thoughts seem to have gone in the same direction as mines... But you seem to have the solution
I will try to check this script at weekend an will let you know if it also works for me.

[tigerA]

From my own experience.
I set: Monitor -> Source -> Ffmpeg. Protocol RTSP.
In Storage tab: Save JPEGs -> Disabled, VideoWriter - > H264 Camera Passtrough

[Pedulla]

Are you getting audio with this too?

[ebeng]

Thanks for the script.

I've had to create a <.mylogin.cnf> file for the MySQL statements, because when cleartext passowrd is entered in the script, you will get warning from MySQL:

Code: Select all

mysql_config_editor set --login-path=local --host=localhost --user=db_user --password

I've replaced all the

Code: Select all

mysql -NBqr zm -e 

to

Code: Select all

mysql --login-path=local -NBqr zm -e 

because of the database for ZoneMinder is with a user/pass.

However, bumped up to following error when testing the zmrecord script:

Code: Select all

root@zoneminder:~# zmrecord.sh 1
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '' at line 1
/usr/local/bin/zmrecord.sh Monitor Data Error. Got: Width 1920 1080 rtsp://xxx:xxx@10.10.10.3:554/cam/realmonitor?channel=1&subtype=0 0 Height  ReadPath  StorePath /1
root@zoneminder:~#

This is when executing the first 2 commands in the script.

Code: Select all

ubuntu@zoneminder:~$ mysql --login-path=local -NBqr zm -e "select Width,Height,Path,StorageId from Monitors where Id=1"
1920    1080    rtsp://xxx:xxx@10.10.10.3:554/cam/realmonitor?channel=1&subtype=0     0
ubuntu@zoneminder:~$ mysql --login-path=local -NBqr zm -e "select Path from Storage where Id=0"
/var/cache/zoneminder/events
ubuntu@zoneminder:~$

So, I did split up the command to get the variables:

Code: Select all

read -r WIDTH <<< `mysql --login-path=local -NBqr zm -e "select Width from Monitors where Id=$MONITOR_ID"`
read -r HEIGHT <<< `mysql --login-path=local -NBqr zm -e "select Height from Monitors where Id=$MONITOR_ID"`
read -r READPATH <<< `mysql --login-path=local -NBqr zm -e "select Path from Monitors where Id=$MONITOR_ID"`
read -r STORAGEID <<< `mysql --login-path=local -NBqr zm -e "select StorageId from Monitors where Id=$MONITOR_ID"`
read -r STORAGE <<< `mysql --login-path=local -NBqr zm -e "select Path from Storage where Id=$STORAGEID"`
STORE_PATH=$STORAGE/$MONITOR_ID

This started the stream to be captured !!

Code: Select all

root      7929  7921  0 20:40 pts/1    00:00:00 ffmpeg -y -loglevel quiet -t 600 -i rtsp://xxx:xxx@10.10.10.3:554/cam/realmonitor?channel=1&subtype=0 -c:v copy /var/cache/zoneminder/events/1/Rec-m4b5nR.mp4

Now i've copied the zmrecord@.service script to the folder:

Code: Select all

root@zoneminder:/lib/systemd/system# systemctl enable zmrecord@1.service zmrecord@4.service zmrecord@5.service zmrecord@6.service
Created symlink from /etc/systemd/system/multi-user.target.wants/zmrecord@1.service to /lib/systemd/system/zmrecord@.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/zmrecord@4.service to /lib/systemd/system/zmrecord@.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/zmrecord@5.service to /lib/systemd/system/zmrecord@.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/zmrecord@6.service to /lib/systemd/system/zmrecord@.service.

when want to start the stream for the ID=1 camera, it still tries to run mysql with the user 'root':

Code: Select all

May 28 22:09:56 zoneminder systemd[1]: Started Record Stream from Camera 1.
May 28 22:09:56 zoneminder zmrecord.sh[3979]: ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)
May 28 22:09:56 zoneminder zmrecord.sh[3979]: message repeated 4 times: [ ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)]
May 28 22:09:56 zoneminder zmrecord.sh[3979]: /usr/local/bin/zmrecord.sh Monitor Data Error. Got: Width  Height  ReadPath  StorePath /1
May 28 22:09:56 zoneminder systemd[1]: zmrecord@1.service: Main process exited, code=exited, status=1/FAILURE
May 28 22:09:56 zoneminder systemd[1]: zmrecord@1.service: Unit entered failed state.
May 28 22:09:56 zoneminder systemd[1]: zmrecord@1.service: Failed with result 'exit-code'.
May 28 22:09:56 zoneminder systemd[1]: zmrecord@1.service: Service hold-off time over, scheduling restart.
May 28 22:09:56 zoneminder systemd[1]: Stopped Record Stream from Camera 1.

It seems to be that the SYSTEMCTL is running with root, I've also defined in the service file + make it start after zoneminder + mysql:

Code: Select all

[Unit]
Description=Record Stream from Camera %I
After=mysql.service ssh.service networking.service zoneminder.service


[Service]
Type=simple
ExecStart=/usr/local/bin/zmrecord.sh %I
Restart=on-failure
User=root

[Install]
WantedBy=multi-user.target

after you did create the <.mylogin.cnf>, just copy this file also to the root. It will work via there.

THX for the script!!!!

With my server, the CPU is like 1-2% now for everything.... before with ZoneMinder for every capture it was like 80%!!

This is how the HIGH-RES is being recorde

Code: Select all

 1489 root      20   0  427940  30608  24020 S   0.3  0.2   0:00.33 ffmpeg
 1488 root      20   0  428004  30516  23900 S   0.3  0.2   0:00.25 ffmpeg
 1492 root      20   0  428000  30484  23840 S   0.3  0.2   0:00.37 ffmpeg
 1487 root      20   0  427948  30312  23712 S   0.3  0.2   0:00.33 ffmpeg

This is for the MODECT... huge difference!!!

Code: Select all

 1954 www-data  20   0 2493396 1.443g 1.177g R  92.7  9.2   1:26.94 zmc
 1887 www-data  20   0 2493352 1.442g 1.177g R  98.3  9.2   2:17.66 zmc
 1783 www-data  20   0 2493220 1.442g 1.177g R  95.0  9.2   4:18.39 zmc
 1922 www-data  20   0 2493488 1.442g 1.177g R 107.0  9.2   1:49.09 zmc
 1964 www-data  20   0 1572148 1.206g 1.173g S  21.6  7.7   0:21.19 zma
 1891 www-data  20   0 1562024 1.196g 1.171g S  21.6  7.6   0:29.87 zma
 1932 www-data  20   0 1562024 1.195g 1.171g S  22.6  7.6   0:21.95 zma
 1803 www-data  20   0 1560004 1.194g 1.172g S  20.6  7.6   0:58.16 zma

[tuxmos]

Thanks for the work. It works so well. I have made adjustments to myself:
Sometimes a stream stops, or stops recording. The service will not restart automatically. You would have to install a watchdoc for it. First of all, I helped to restart all stream services once an hour. Unfortunately, it always remains an mp4 corpse ($ tmpfile) in the root folder of the camera. I inserted a "rm $ STORE_PATH / *. Mp4" in zmrecord.sh after "STORE_PATH = $ STORAGE / $ MONITOR_ID".

Furthermore, all new events in this recording are now called "Event-NUMBER", regardless of "EventPrefix". That would have to be corrected. Is very helpful in filtering the events.
EDIT:
Insert after read Parameters:
read -r EPREFIX <<< `mysql -NBqr ZM -e "select EventPrefix from Monitors where Id=$MONITOR_ID"`
and change "Event-" in UPDATE Events SET Name=CONCAT(... to "$EPREFIX"

What will probably be even more difficult, is that the whole with rotated cameras does not really work / can work. Or? Can ffmpeg also pass the value "Orientation", so that e.g. a camera that is 90 ° rotated, so that a high-profile video is recorded correctly?

[burleyj]

Wow what a great post. I came across this on accident and wanted to try it out. I had the same problem with the query that pulls the Monitor information - for some reason the whole output was assigned to the first variable Width. From everything I read it seems that the \t separator should have worked but it didn't. I'm using Ubuntu 16.04 LTS.

Even after changing that, it kept failing to run, giving me errors like the stream file wasn't being created.

user@ubuntu:/mnt/cameras/4$ sudo /usr/local/bin/zmrecord.sh 4
/usr/local/bin/zmrecord.sh: line 62: mediainfo: command not found
stat: cannot stat '/mnt/cameras/4/Rec-POtUts.mp4': No such file or directory
/usr/local/bin/zmrecord.sh: line 62: mediainfo: command not found
/usr/local/bin/zmrecord.sh: line 62: mediainfo: command not found

After a while I noticed the -loglevel quiet in the script and removed it. When running the script the next time, I got this error.

[aac @ 0x1e201a0] The encoder 'aac' is experimental but experimental codecs are not enabled, add '-strict -2' if you want to use it.

I tried adding it but it still didn't work. Then I came across some info about it being important where the switch was added.
https://stackoverflow.com/questions/329 ... ot-enabled
That thread showed that it needs to be the last option before the output file.

Code: Select all

ffmpeg -y -loglevel quiet -t $RECORD_TIME -i "$READPATH" -c:v copy -strict -2 $tmpfile

So now it works for me! Well it's creating the file anyway, I haven't gotten past that yet. Calling it a night soon and didn't want to forget to add to this excellent post. Hopefully it helps someone else who may have the same problem. My camera is the LaView One Halo.

[mwvent]

Thank you so much for posting this! I was shocked when I got some new 1080p cams that Zoneminders pass-through solution is so CPU intensive! I think this would be great as a built in monitor mode

I have made a few edits to take advantage of FFMPEG's segment output to get gapless recording and also added the ability to add the segments to the DB as they are being wrote to disk. I am able to watch the events in the web interface before they have finished writing and get current FPS on the console.

https://gist.github.com/mwvent/f4d30599 ... 5ea1fb4c76

Hope it helps and thank you again for posting this solution.

[tuxmos]

Oh great. Thanks for the progress. Two things: 1. An error in the script, maybe? Line 45: MESSAGE = "` echo $ 3 | tr -d '"'` "is shown to me as a syntax error. The correct thing would be MESSAGE =` echo $ 3 | tr -d '"' 'and secondly: can you please include EPREFIX, as I described in my previous post?

/ SNIP: Furthermore, all new events in this recording are now called "Event-NUMBER", regardless of "EventPrefix". That would have to be corrected. Is very helpful in filtering the events.
EDIT:
Insert after read parameters:
read -r EPREFIX <<< `mysql -NBqr ZM -e" select EventPrefix from Monitors where Id = $ MONITOR_ID "`
and change "Event-" in UPDATE Events SET Name = CONCAT (... to "$ EPREFIX"
/ SNAP

[russell_i_brown]

Very nice mods by Matt - Thanks muchly.

I've just installed a shiny new ZM 1.34.1 system and notice an issue with the StorageId (it seems to be 0 on new monitors but the Storage Db index starts at 1).

Anyway, I fixed that in zmrecord.sh and added support for EventPrefix for tuxmos.

While furtling, I also put in some calcs to produce Bandwidth numbers so the ZM console looks sane and the total BW includes these streams.

My fork (version 0.7) is here: https://gist.github.com/ruffle-b/5ab39b ... 678698bf5f

but I've told Matt that I'm more that happy for him to merge them and call his Gist the master. If he does that then I'll delete the above Gist and Matt's will be at https://gist.github.com/mwvent/f4d30599 ... 5ea1fb4c76

[tuxmos]

Thank you for the implementation. Unfortunately, version 0.7 causes a significantly higher system load than my modified version 0.3, as described above. I have 9 HD streams with this script in the recording and then as many SD streams with Modect. The system load is approximately 10% for all four cores. With version 0.7, the system load on each core increases to over 50% and higher and the scripts themselves appear at the top in HTOP. So I can not say what exactly triggers the higher system load.

[russell_i_brown]

That's weird. My setup's much the same as yours and zmrecord.sh v0.7 doesn't go above ~3% load for a 2688x1520x32 camera.

Does going from 0.3 to 0.6 cause the leap in load or is it only 0.7 ?

[tuxmos]

Hi
I have not tried the versions in between.
In the meantime, I had to completely rebuild the server due to a hard disk error and am now using version 0.7. CPULoad is still around 50% on all four cores. When testing the new server, I noticed a lot. e.g. Version 0.7 with streams with audio enabled in the camera seems to have a problem.
If I turn off audio in the camera, everything is ok.
The generated events have all 0 bytes and a length of 0. Furthermore, the following sometimes appears: "ERROR 1366 (HY000) at line 1: Incorrect decimal value: '' for column 'CaptureFPS' at row 1". Tested in the console with zmrecord.sh <MonitorId>



Here is a screenshot of Htop on the server

[russell_i_brown]

I don't have any cameras with audio capability... which will make replicating your issue tricksy

However, this does sound like an ffmpeg related problem. The output files should not be 0 bytes. The MySQL error about CaptureFPS also points towards ffmpeg as the status output from ffmpeg is what's being used for that value.

Try "cat /dev/shm/zmrecord-m*-stats.txt" and you'll see the current ffmpeg status for each stream. Mine looks like:

Code: Select all

russell@cctv:~$ cat /dev/shm/zmrecord-m*-stats.txt
1583346960 11.99 N/A 5213318
1583346960 16.56 N/A 7199700
1583346960 16.00 N/A 6943297
1583346960 16.08 N/A 6991434
1583346960 11.99 N/A 5213300
1583346960 10.00 N/A 4345012
1583346960 12.00 N/A 5217988
1582908548 12.01 N/A 289669
russell@cctv:~$ 

The second field is the FPS value which is what's being injected (and complained about) in MySQL.

FWIW, I'm running eight low-res modect monitors and seven high res (up to 4K) zmrecord streams. This is on a 4 core E3-1220 Xeon and my
load on the fully running system is like so:



I am running a GPU which cuts down the load generated by zmc but only by about half and the ffmpeg fired up by zmrecord doesn't use the GPU at all.

Try firing up zmrecord, see what the ffmpeg command line is, cut'n'paste and run in a shell - you should see a stream of output like:

Code: Select all

frame=36
fps=0.00
stream_0_0_q=-1.0
bitrate=N/A
total_size=N/A
out_time_us=2423421
out_time_ms=2423421
out_time=00:00:02.423421
dup_frames=0
drop_frames=0
speed= 4.1x
progress=continue
frame=43
fps=36.09
stream_0_0_q=-1.0
bitrate=N/A
total_size=N/A
out_time_us=3023438
out_time_ms=3023438
out_time=00:00:03.023438
dup_frames=0
drop_frames=0
speed=2.54x
progress=continue
frame=50

Perhaps that will yield a clue.

PS - RAID is a really really Good Idea