High resolution performance - need help

[mattj]

I just built a new very fast machine, and it cannot encode frames fast enough at higher resolutions. Is there anything I can do to improve it?

At 1920p (2592x1920) 32-bit color with the camera set to 15fps, I'm actually only getting about 2fps both recorded and in the live monitor. CPU load is between 2.5 and 3.1, the camera is streaming via RTP/RTSP and ffmpeg, 32-bit color. Enabling or disabling motion detection does not seem to change the fps. The live monitor shows the same frame rate as what is recorded. The problem improves as I lower the resolution, but I'm all the way down to 1080p before I get anything close to 15fps recorded. There are only 2 cameras on the machine, one at 920p 7fps (MicroSeven) and one at 1920p 15fps (5MP).


The machine is a 6-core i7 @3.3GHz, 16GB quad channel DDR4 RAM 2666MHz, pair of Western Digital Se hard disks. Linux shows 12 cores available, CPU load varies between about 2.5 and 3.1. The log below is of me setting the ZM state to "Running" then walking in front of the 1920p camera to cause a motion event. I tried to attach it as a file but the forum seems to not allow me to attach a .txt or .log file.

Any tips would be greatly appreciated!



-------------------------------------


Date/Time
Component Server PID Level Message File Line
2016-07-24 08:55:32.285427 zma_m1 7994 INF MicroSeven: 1000 - Analysing at 6.94 fps zm_monitor.cpp 1287
2016-07-24 08:55:29.832510 zmc_m1 7957 INF MicroSeven: 1000 - Capturing at 6.90 fps zm_monitor.cpp 3131
2016-07-24 08:55:19.176225 zmc_m2 7956 INF 5MP: 2000 - Capturing at 14.93 fps zm_monitor.cpp 3131
2016-07-24 08:54:53.866361 zma_m2 7995 INF 5MP: 1000 - Analysing at 9.52 fps zm_monitor.cpp 1287
2016-07-24 08:54:21.637180 zms 8022 INF Got signal 15 (Terminated), exiting zm_signal.cpp 40
2016-07-24 08:54:12.510485 zmc_m2 7956 INF 5MP: 1000 - Capturing at 14.71 fps zm_monitor.cpp 3131
2016-07-24 08:54:08.560331 undef 8022 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:12.850037 zma_m1 7994 INF MicroSeven: 026 - Opening new event 131, section start zm_monitor.cpp 1532
2016-07-24 08:53:10.425828 zma_m1 7994 INF In mode 5/1, warming up zma.cpp 142
2016-07-24 08:53:09.369771 zma_m1 7994 WAR Waiting for capture daemon zm_monitor.cpp 503
2016-07-24 08:53:08.978599 zma_m2 7995 INF 5MP: 026 - Opening new event 130, section start zm_monitor.cpp 1532
2016-07-24 08:53:08.578349 zma_m2 7995 INF In mode 5/1, warming up zma.cpp 142
2016-07-24 08:53:08.450862 undef 7995 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:08.329830 zmdc 7995 INF 'zma -m 2' started at 16/07/24 08:53:08 zmdc.pl
2016-07-24 08:53:08.329780 zmdc 7901 INF 'zma -m 2' starting at 16/07/24 08:53:08, pid = 7995 zmdc.pl
2016-07-24 08:53:08.302338 zma_m1 7994 WAR Waiting for capture daemon zm_monitor.cpp 503
2016-07-24 08:53:08.236090 zmdc 7901 INF Starting pending process, zma -m 2 zmdc.pl
2016-07-24 08:53:08.230874 undef 7994 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:08.146220 zmdc 7994 INF 'zma -m 1' started at 16/07/24 08:53:08 zmdc.pl
2016-07-24 08:53:08.146210 zmdc 7901 INF 'zma -m 1' starting at 16/07/24 08:53:08, pid = 7994 zmdc.pl
2016-07-24 08:53:08.035150 zmdc 7901 INF Starting pending process, zma -m 1 zmdc.pl
2016-07-24 08:53:04.759958 zmc_m2 7956 INF Priming capture from rtsp://192.168.1.86:554/h264?username=admin&password=demo321 zm_ffmpeg_camera.cpp 104
2016-07-24 08:53:04.733101 zmc_m2 7956 INF Starting Capture version 1.29.0 zmc.cpp 250
2016-07-24 08:53:04.517810 zmwatch 7963 INF Watchdog pausing for 30 seconds zmwatch.pl
2016-07-24 08:53:04.517799 zmc_m1 7957 INF Priming capture from rtsp://root:yz80yz80microseven!!@192.168.1.83:554/11 zm_ffmpeg_camera.cpp 104
2016-07-24 08:53:04.408780 zmwatch 7963 INF Watchdog starting zmwatch.pl
2016-07-24 08:53:04.347522 zmc_m1 7957 INF Starting Capture version 1.29.0 zmc.cpp 250
2016-07-24 08:53:04.293200 zmdc 7963 INF 'zmwatch.pl' started at 16/07/24 08:53:04 zmdc.pl
2016-07-24 08:53:04.293190 zmdc 7901 INF 'zmwatch.pl' starting at 16/07/24 08:53:04, pid = 7963 zmdc.pl
2016-07-24 08:53:04.186990 zmfilter 7941 INF Scanning for events zmfilter.pl
2016-07-24 08:53:04.152529 undef 7957 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:04.069270 zmdc 7958 INF 'zmaudit.pl -c' started at 16/07/24 08:53:04 zmdc.pl
2016-07-24 08:53:04.069250 zmdc 7901 INF 'zmaudit.pl -c' starting at 16/07/24 08:53:04, pid = 7958 zmdc.pl
2016-07-24 08:53:04.010760 zmdc 7901 INF 'zmc -m 1' starting at 16/07/24 08:53:04, pid = 7957 zmdc.pl
2016-07-24 08:53:04.010760 zmdc 7957 INF 'zmc -m 1' started at 16/07/24 08:53:04 zmdc.pl
2016-07-24 08:53:03.933530 zmdc 7901 INF Starting pending process, zmc -m 1 zmdc.pl
2016-07-24 08:53:03.914356 undef 7956 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:03.875170 zmdc 7901 ERR 'zma -m 1' exited abnormally, exit status 255 zmdc.pl
2016-07-24 08:53:03.759080 zmdc 7956 INF 'zmc -m 2' started at 16/07/24 08:53:03 zmdc.pl
2016-07-24 08:53:03.759070 zmdc 7901 INF 'zmc -m 2' starting at 16/07/24 08:53:03, pid = 7956 zmdc.pl
2016-07-24 08:53:03.707936 zma_m1 7955 ERR Shared data not initialised by capture daemon for monitor MicroSeven zm_monitor.cpp 442
2016-07-24 08:53:03.656880 zmdc 7901 INF Starting pending process, zmc -m 2 zmdc.pl
2016-07-24 08:53:03.636723 undef 7955 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:03.598390 zmdc 7901 ERR 'zma -m 2' exited abnormally, exit status 255 zmdc.pl
2016-07-24 08:53:03.466820 zmdc 7955 INF 'zma -m 1' started at 16/07/24 08:53:03 zmdc.pl
2016-07-24 08:53:03.466790 zmdc 7901 INF 'zma -m 1' starting at 16/07/24 08:53:03, pid = 7955 zmdc.pl
2016-07-24 08:53:03.374064 zma_m2 7954 ERR Shared data not initialised by capture daemon for monitor 5MP zm_monitor.cpp 442
2016-07-24 08:53:03.297910 zmdc 7901 INF Starting pending process, zma -m 1 zmdc.pl
2016-07-24 08:53:03.278659 undef 7954 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:03.166330 zmdc 7901 INF 'zma -m 2' starting at 16/07/24 08:53:03, pid = 7954 zmdc.pl
2016-07-24 08:53:03.166330 zmdc 7954 INF 'zma -m 2' started at 16/07/24 08:53:03 zmdc.pl
2016-07-24 08:53:03.105900 zmdc 7901 INF Starting pending process, zma -m 2 zmdc.pl
2016-07-24 08:53:03.022430 zmdc 7901 ERR 'zma -m 2' exited abnormally, exit status 255 zmdc.pl
2016-07-24 08:53:02.947370 zmdc 7901 ERR 'zma -m 1' exited abnormally, exit status 255 zmdc.pl
2016-07-24 08:53:02.905540 zmdc 7901 ERR 'zmc -m 2' exited abnormally, exit status 255 zmdc.pl
2016-07-24 08:53:02.826570 zmdc 7901 ERR 'zmc -m 1' exited abnormally, exit status 255 zmdc.pl
2016-07-24 08:53:02.822962 zma_m1 7929 ERR Got empty memory map file size 0, is the zmc process for this monitor running? zm_monitor.cpp 533
2016-07-24 08:53:02.781049 zma_m2 7937 ERR Got empty memory map file size 0, is the zmc process for this monitor running? zm_monitor.cpp 533
2016-07-24 08:53:02.767369 undef 7929 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:02.767321 undef 7933 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:02.730737 undef 7925 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:02.729723 undef 7937 INF No Server ID or Name specified in config. Not using Multi-Server Mode. zm_config.cpp 139
2016-07-24 08:53:02.729370 zmdc 7901 INF 'zmfilter.pl' starting at 16/07/24 08:53:02, pid = 7941 zmdc.pl
2016-07-24 08:53:02.729370 zmdc 7941 INF 'zmfilter.pl' started at 16/07/24 08:53:02 zmdc.pl
2016-07-24 08:53:02.441510 zmdc 7937 INF 'zma -m 2' started at 16/07/24 08:53:02 zmdc.pl
2016-07-24 08:53:02.441130 zmdc 7901 INF 'zma -m 2' starting at 16/07/24 08:53:02, pid = 7937 zmdc.pl
2016-07-24 08:53:02.187840 zmdc 7933 INF 'zmc -m 2' started at 16/07/24 08:53:02 zmdc.pl
2016-07-24 08:53:02.187820 zmdc 7901 INF 'zmc -m 2' starting at 16/07/24 08:53:02, pid = 7933 zmdc.pl
2016-07-24 08:53:01.924500 zmdc 7929 INF 'zma -m 1' started at 16/07/24 08:53:01 zmdc.pl
2016-07-24 08:53:01.924440 zmdc 7901 INF 'zma -m 1' starting at 16/07/24 08:53:01, pid = 7929 zmdc.pl
2016-07-24 08:53:01.704340 zmdc 7901 INF 'zmc -m 1' starting at 16/07/24 08:53:01, pid = 7925 zmdc.pl
2016-07-24 08:53:01.704320 zmdc 7925 INF 'zmc -m 1' started at 16/07/24 08:53:01 zmdc.pl
2016-07-24 08:53:01.519410 zmpkg 7885 INF Starting up services zmpkg.pl
2016-07-24 08:52:58.515560 zmdc 7901 INF Server starting at 16/07/24 08:52:58 zmdc.pl
2016-07-24 08:52:58.220430 zmpkg 7885 INF Command: start zmpkg.pl
2016-07-24 08:52:58.177590 zmpkg 7885 INF Sanity checking States table... zmpkg.pl
2016-07-24 08:52:58.003180 zmsystemctl 7876 INF Redirecting command through systemctl zmsystemctl.pl
2016-07-24 08:52:57.770180 zmpkg 7870 INF Command: start zmpkg.pl
2016-07-24 08:52:57.726260 zmpkg 7870 INF Sanity checking States table... zmpkg.pl

[bbunge]

Linux distro and version?

[mattj]

Ubuntu server, the latest stable release. I'll get the exact version next time I'm in front of the computer. It's a fresh install.

[mattj]

$ uname -a
Linux ZoneMinder 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Installed from: ubuntu-16.04-server-amd64.iso


Thanks!

[bbunge]

You may be running out of space on tmpfs. Lower the frame rate to 5 and resolution

[knight-of-ni]

This appears to be a followup to this thread:
viewtopic.php?f=34&t=25044

mattj, I'm not sure you I know what to say. In the thread I just linked, I gave you a recommendation on what kind of hardware could support the cameras you described. You chose to ignore that recommendation. That is certainly your choice, but here you are asking for additional help from us to troubleshoot hardware that is not fast enough to handle the load you are giving it.

Your i7 cpu is fast, but like I mentioned before more cores give you much more bang for your buck. Hyperhtreaded cores don't count.
You've only got 2 hard drives and have not described your RAID configuration so I'm going to assume your system is essentially recording to one disk at a time. That simply won't be fast enough. What I recommend you do is add a smaller, dedicated hard drive to boot from. It does not have to be super fast. Then take those two large drives, and put them in RAID 0 for your video event storage. RAID 0 has no redundancy, however.

Once you do that, follow the advice bbunge just told you and slow your frame rate down to 5. Now, disable all your cameras but one. Then slowly enable your cameras, very slowly one at a time, while keeping an eye on the space in tmpfs (this is your ram disk which zoneminder uses a lot of for shared memory). Also watch cpu load, overall memory consumed, and disk I/O.

[knight-of-ni]

Almost forgot... when you lower the frame rate, do it in the camera. Do NOT try to do it in ZoneMinder.

[mattj]

Thanks guys for the replies. My tmpfs on /run has 1.7GB free. The other tempfs mounts have nothing used.

@knnniggett - I definitely did not ignore your hardware recommendations. In this case I've only got 2 cameras running, one at a relatively low resolution of only 920p, so I don't think it's a lack of processing power or hard disk speed. I get the same exact results when I reduce it to only the single 5MP camera at 15fps. This means with the hard disk and 6 processor cores dedicated to a single camera, I'm still only getting a couple of fps. I don't think it's due to lack of total processing power or disk speed, but I do get the impression that a lot of something is happening in a single thread and it's not taking advantage of all available cores. So either I need to make something more efficient, or I'm just not going to be able to run high res.

My drive configuration is non-RAID with a dedicated 1TB boot disk, and two 4TB drives in a volume group for 8TB total event storage. With the single camera running, the disks seem to be mostly idle. I am setting the frame rate at the camera.

Has anyone else tried running a camera in 5MP resolution (1920p), what what were your results?

I do appreciate the help from everyone, this is as much a learning experience as it is me trying to attain a goal.

[knight-of-ni]

OK, fair enough. My bad. If you are only running two cameras, you should be able to get things working. It is unlikely you will get all 12 to work at full frame rate, but we can work up to that to figure out some reasonable values for your system. Once you get all your cameras online and writing to disk, it is also unlikely a single drive will be able to handle the load. It is easier to reconfigure those drives for RAID 0 now, rather than later when you've got 4TB of video events on the disk.

Note that ZoneMinder is hard coded to a maximum resolution of 2048x1536 which works out to 3MP:
https://github.com/ZoneMinder/ZoneMinde ... g.h.in#L39

1920x1080 will work, but not 5MP, 2592 x 1920.

Can you post your complete monitor configuration when running at 1920p? We need to see the General and Source tabs. Screenshots are preferred.
With just a single camera running at 5fps, can you post a new set of logs? We need to see what shows up in the logs from the moment you start zoneminder to just after you open a live monitor.

Also, with a single cameras running, take a close look at how much cpu the zm* processes are using, how big your /dev/shm is, how much physical ram is free, and continue to watch disk i/o when zoneminder writes to disk. We need to establish a baseline for a single camera.

Background:
ZoneMinder was written in 2003, when the majority of surveillance systems used capture cards. VGA resolution was the maximum resolution and frame rates were low. With those parameters, recording events as a lot of static jpeg's was reasonable. Move ahead to today and the core of ZoneMinder has not changed much. Overtime, the ability to connect to IP cameras was piggy-backed on, but the core remains the same. This presents a problem for large resolutions and frame rates, and that is what you are fighting against. We have a feature branch which writes to mp4 containers, but that is still experimental. Last time I checked it was still not ready for production. You are welcome to help test that if you are willing to be a guinea pig.

[mattj]

Thanks for the help. We just purchased our new house yesterday, where the cameras are going to be installed, so this week will be 24/7 craziness as we repair to so we can move in this coming weekend. I will post all the info that you ask for as soon as I get 10 minutes in front of the computer to collect it.

[knight-of-ni]

Thanks for letting us know, and be careful not to fall through the ceiling of your new house while you are running wires for your cameras, unless you intend to work on your drywall skills. I've done that.