[SOLVED] Monitoring high resolution/high fps feed

[Uschiekid]

I am using Ubuntu 14.04LTS, with ZoneMinder 1.30.2.

I used the easy install method (then upgraded using ppa Zoneminder-master, as only 1.26.5 was available on stable, for 14.04) EDIT: I realise now that 1.30.0 was only temporarily not available for 14.04 when I was trying to get a current version, with only the software center version available at the time. When I check the stable ppa now, trusty is there with 1.30.0. I have stopped subscribing to the zoneminder-master ppa, so will just get upgrades from the stable ppa from now. However, this problem existed with my previous install that was a recent version, possibly 1.30.0, or the previous version to that on the stable ppa. The reinstall had to do with an ubuntu issue, where I had to reinstall many things, not specifically to do with the issue listed below. END OF EDIT.

I have got various cameras and setups to work without issue, but I am not an expert, so I can't get rid of this one issue.

My intended use is to VIEW ONLY, with no motion detection or recording needed (I will do that through other means).
My understanding is that this should be done with the "Monitor" function. I don't need the pause and rewind functions, I just want to view the feed live. I want to have the monitor window open during waking hours. This works, with one caveat.

This concerns my main camera, a foscam R2 (V3) (running as a remote rtsp/unicast (EDIT: same results with remote rtp/rtsp and Ffmpeg rtp/rtsp, btw)--other values in monitor setup as default), which I want to use high resolution for my viewing (sometimes 720p and sometimes 1080p), and I believe my computer can handle high fps, since it works with both VLC and Xeoma. The camera's Mainstream is set at 1080p/25fps and the Substream is 720p/11fps. When I initiate a viewing window, all runs smoothly.

Then (based on logging of the capture) after 2000 captured frames or so, the problem starts.

I have a constant system monitor on my gnome panel, and I see that moderate activity is being written to disk. Then after 3000 captured frames, this becomes worse and so on. It ends up having 60% or more of the disk in use, for minutes at a time. Which is obviously not a good thing. And of course using the disk constantly ends up causing the computer to heat up over time.

I believe this is because there is not enough memory being used, and so it is writing to disk instead. But I have 16GB memory, as well as the same in SWAP, even when most is not used, this still occurs. I've hunted for a solution, and the closest thing that I've stumbled upon that might affect this is the setting FRAME_SOCKET_SIZE, but I can't find any example values for when the default 0 doesn't work. It suggests:
"You can use
this option to indicate that a larger buffer size should be used. Note
that you may have to change the existing maximum socket buffer size on
your system via sysctl (or in /proc/sys/net/core/wmem_max) to allow
this new size to be set"

But I really don't know enough about this to mess around with the system settings without advice, even though I tried to read up on it. So I don't want to experiment, unless I know that this definitely is the issue, or at least what values might be good to try.

The camera itself works, when this problem shows up, Zoneminder doesn't produce any errors, just states something like "2017-02-16 20:55:34.020023 zmc_m13 12095 INF HumLANSub: 6000 - Capturing at 11.36 fps zm_monitor.cpp 3125" every few minutes.
Looking through my log shows lots of errors, but they are from when I was trying to fix this by changing settings, and then changing back. If I restart my zoneminder, and then just view a camera, I don't think it has any errors to show something is going wrong. If someone wants me to perform certain actions, and then post the log, I can do that.

If the above setting is relevant, my cat /proc/sys/net/core/wmem_max is 212992

If any one can give me any advice, suggestions or help, I would really appreciate it. I want to get this camera viewing smoothly before the return period is up. I'm not a linux expert, but I do use Ubuntu pretty much exclusively.

Thanks in advance

Edit: There are some warnings and errors when starting up the monitor, but while viewing, there are only the logs listed above. However I'm attaching a log showing the errors and warnings as well. This is with the R2 camera @720p/11fps and a low fps/620x480 res android phone turned ip cam (both in a dark room, so no activity) . Each with a window open, in monitoring state.

zm log viewing.txt

(5.24 KiB) Downloaded 298 times

[Uschiekid]

Possible solution?

I am not very knowledgeable about this AT ALL, so excuse any dumb/noob-nesses

My /tmp is not tmpfs, from what I can tell. I wonder if this is why I am seeing so much disk activity. This article describes changing /tmp to tmpfs
https://insights.ubuntu.com/2016/01/20/ ... -on-tmpfs/

using:

Code: Select all

$ echo "tmpfs /tmp tmpfs rw,nosuid,nodev" | sudo tee -a /etc/fstab
$ sudo reboot

I know this isn't specifically Zoneminder, but could this stop my Monitor viewing writing to disk?

On a side note, I have tried using low res/low fps (640x360 and 10fps) and high res/low fps (1080p and 5fps), this problem still exists, although the higher res and fps means it starts almost immediately.

EDIT: I did also try the temporary change in swappiness listed on this ZM wiki:
https://wiki.zoneminder.com/Common_Issu ... _on_Ubuntu

It didn't seem to have an effect, which didn't surprise me as my SWAP isn't being used anyway.

Here's an image of my system monitor, just at the start of the problem...at that point, the monitor had been open for a few minutes. Disk in orange, the empty one is swap, green is memory.

[Uschiekid]

The method listed above changing /tmp to tmpfs seems to have worked! No more writing to disk. The cpu usage goes up, but I think I can live with that.

[gazooz]

I have the Foscam r2 as well. I can't seem to get the PTZ working right. i'm close, as it does move the camera, however the other controls don't work as expected, one changes saturation, another the brightness and panning left or right don't stop until the cam is full 180 degrees. were you able to get yours working correctly? any help is greatly appreciated.

[pgrunwald]

Instead of changing /tmp did you try changing PATH_SWAP to /dev/shm?

[pgrunwald]

I have been getting intermittent bursts of errors like this.

ERR [Incomplete frame write: 219264 of 289380 bytes written]

my cat /proc/sys/net/core/wmem_max was also 212992. Last night I took my high res camera to 24-bits and set net.core.wmem_max = 262144 in /etc/sysctl.conf.
I put this also in FRAME_SOCKET_SIZE

(262144 = 256k)

PATH_SWAP to /dev/shm was already set.

This was a good reference: https://www.cyberciti.biz/faq/linux-tcp-tuning/

My logs have been clean for 24-hours of this error and I added second another camera tonight with no errors!

Zoneminder 1.30.0 on "Debian 9 64-bit with Zoneminder 1.30.0 the Easy Way (alternative)
Hardware is Dell R710 using https://debian.pkgs.org/8/debian-nonfre ... l.deb.html

Using both feeds from each camera (Reolink RLC410 & 411)

low res
FFMPEG
modect
analysis fps 5
max fps 9 (camera at 7)
rtsp://xxxx:xxxxx@192.168.1.102:554//h264Preview_01_sub
RTP/RSTP
24-bit color
640 x 360

high res
FFMPEG
nodect (triggered on low res)
analysis fps 1
max fps 21 (camera at 20)
rtsp://xxxx:xxxxx@192.168.1.102:554//h264Preview_01_main
RTP/RSTP
24-bit color
2304 x 1294

edit:
I added the following to /etc/sysctl.conf

net.core.wmem_max=262144
net.core.rmem_max=262144
net.core.optmem_max=262144

then did sysctl -p.

[apbb2]

I just put in a replacement install for an old system because /dev/shm was at 100% on old system & did not pay to upgrade.

I have been replacing all of our ZM server where we cannot use VM with Intel I7 NUC with 16gb RAM & 1tb SSD & has been rocK solid on all 6 other installs. On this last one, I had high CPU load (6.50-7.20) & could not understand why. Changed PATH_SWAP TO /dev/shm & I watched it jumped down & settle into 1.14 area.

This was an easy fix for me, thanks.

BTW, for max mem I use a little script I found here a while ago & have been using since 1.24 or 1.25. It was called pada.sh. Easy way to setup sysctl without having to do any calculations. It is always part of my install.

Start with sudo nano pada.sh

Cut and paste quote below;

#!/bin/sh
# ZoneMinder 1.24.2 installation script
#
# Copyright (C) 2010 Chris "Pada" Kistner
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.

echo "--- Checking for root privileges..."
if [ "`whoami`" != "root" ]; then
echo Error: This script requires root access
exit 1
fi

echo "--- Installing extra codecs, which includes x264 support..."
apt-get -y install libavcodec-extra-52 libavformat-extra-52 libswscale-extra-0 libavfilter-extra-1

echo "--- Installing ZoneMinder..."
apt-get -y install zoneminder

echo "--- Adding ZoneMinder to Apache2..."
ln -s /etc/zm/apache.conf /etc/apache2/sites-enabled/zoneminder.conf
service apache2 restart

echo "--- Adding alias for zoneminder service and fixing ZoneMinder startup..."
ln -s /etc/init.d/zoneminder /etc/init.d/zm
update-rc.d -f zoneminder remove
update-rc.d -f zm remove
update-rc.d zm defaults 92
service zm restart

echo "--- Correcting SHM to 90% of memory..."
apt-get -y install coreutils bc
page_size=$(getconf PAGE_SIZE)
mem_bytes=$(awk '/MemTotal:/ { printf "%0.f",$2 * 1024}' /proc/meminfo)
mb=1048576
mem_bytes_mb=$(expr $mem_bytes / $mb)
shmmax=$(echo "$mem_bytes * 0.90" | bc | cut -f 1 -d '.')
shmmax_mb=$(expr $shmmax / $mb)
shmall=$(expr $mem_bytes / $page_size)
shmmax_cur=$(sysctl -n kernel.shmmax)
shmmax_cur_mb=$(expr $shmmax_cur / $mb)
shmall_cur=$(sysctl -n kernel.shmall)
echo "-- Total memory = $mem_bytes B = $mem_bytes_mb MB"
echo "-- Page size = $page_size B"
echo "-- Current kernel.shmmax = $shmmax_cur B = $shmmax_cur_mb MB"
echo "-- Current kernel.shmall = $shmall_cur pages"
if [ "$shmmax" -eq "$shmmax_cur" ] && [ "$shmall" -eq "$shmall_cur" ]; then
echo "-- Recommended shm values already set"
else
echo "-- Recommended: kernel.shmmax = $shmmax B = $shmmax_mb MB"
echo "-- Recommended: kernel.shmmall = $shmall pages"
file=/etc/sysctl.conf
if [ "`grep "^kernel\.shmmax" $file -c`" != "0" ]; then
echo "-- Replacing: kernel.shmmax in $file"
sed "s/^kernel\.shmmax.*$/kernel.shmmax=$shmmax/g" -i $file
else
echo "-- Adding: kernel.shmmax to $file"
echo kernel.shmmax=$shmmax >> $file
fi
if [ "`grep "^kernel\.shmall" /etc/sysctl.conf -c`" != "0" ]; then
echo "-- Replacing: kernel.shmall in $file"
sed "s/^kernel\.shmall.*$/kernel.shmall=$shmall/g" -i $file
else
echo "-- Adding: kernel.shmall to $file"
echo kernel.shmall=$shmall >> $file
fi
echo "-- Using: new sysctl values"
sysctl -p
fi

# Done
echo "--- Done."

Ctrl-w

Ctrl-x

Make it executable:

chmod 777 pada.sh

Then run it:

./pada.sh

[bbunge]

I just put in a replacement install for an old system because /dev/shm was at 100% on old system & did not pay to upgrade.

I have been replacing all of our ZM server where we cannot use VM with Intel I7 NUC with 16gb RAM & 1tb SSD & has been rocK solid on all 6 other installs. On this last one, I had high CPU load (6.50-7.20) & could not understand why. Changed PATH_SWAP TO /dev/shm & I watched it jumped down & settle into 1.14 area.

This was an easy fix for me, thanks.

BTW, for max mem I use a little script I found here a while ago & have been using since 1.24 or 1.25. It was called pada.sh. Easy way to setup sysctl without having to do any calculations. It is always part of my install.

Start with sudo nano pada.sh

Cut and paste quote below;

#!/bin/sh
# ZoneMinder 1.24.2 installation script
#
# Copyright (C) 2010 Chris "Pada" Kistner
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.

echo "--- Checking for root privileges..."
if [ "`whoami`" != "root" ]; then
echo Error: This script requires root access
exit 1
fi

echo "--- Installing extra codecs, which includes x264 support..."
apt-get -y install libavcodec-extra-52 libavformat-extra-52 libswscale-extra-0 libavfilter-extra-1

echo "--- Installing ZoneMinder..."
apt-get -y install zoneminder

echo "--- Adding ZoneMinder to Apache2..."
ln -s /etc/zm/apache.conf /etc/apache2/sites-enabled/zoneminder.conf
service apache2 restart

echo "--- Adding alias for zoneminder service and fixing ZoneMinder startup..."
ln -s /etc/init.d/zoneminder /etc/init.d/zm
update-rc.d -f zoneminder remove
update-rc.d -f zm remove
update-rc.d zm defaults 92
service zm restart

echo "--- Correcting SHM to 90% of memory..."
apt-get -y install coreutils bc
page_size=$(getconf PAGE_SIZE)
mem_bytes=$(awk '/MemTotal:/ { printf "%0.f",$2 * 1024}' /proc/meminfo)
mb=1048576
mem_bytes_mb=$(expr $mem_bytes / $mb)
shmmax=$(echo "$mem_bytes * 0.90" | bc | cut -f 1 -d '.')
shmmax_mb=$(expr $shmmax / $mb)
shmall=$(expr $mem_bytes / $page_size)
shmmax_cur=$(sysctl -n kernel.shmmax)
shmmax_cur_mb=$(expr $shmmax_cur / $mb)
shmall_cur=$(sysctl -n kernel.shmall)
echo "-- Total memory = $mem_bytes B = $mem_bytes_mb MB"
echo "-- Page size = $page_size B"
echo "-- Current kernel.shmmax = $shmmax_cur B = $shmmax_cur_mb MB"
echo "-- Current kernel.shmall = $shmall_cur pages"
if [ "$shmmax" -eq "$shmmax_cur" ] && [ "$shmall" -eq "$shmall_cur" ]; then
echo "-- Recommended shm values already set"
else
echo "-- Recommended: kernel.shmmax = $shmmax B = $shmmax_mb MB"
echo "-- Recommended: kernel.shmmall = $shmall pages"
file=/etc/sysctl.conf
if [ "`grep "^kernel\.shmmax" $file -c`" != "0" ]; then
echo "-- Replacing: kernel.shmmax in $file"
sed "s/^kernel\.shmmax.*$/kernel.shmmax=$shmmax/g" -i $file
else
echo "-- Adding: kernel.shmmax to $file"
echo kernel.shmmax=$shmmax >> $file
fi
if [ "`grep "^kernel\.shmall" /etc/sysctl.conf -c`" != "0" ]; then
echo "-- Replacing: kernel.shmall in $file"
sed "s/^kernel\.shmall.*$/kernel.shmall=$shmall/g" -i $file
else
echo "-- Adding: kernel.shmall to $file"
echo kernel.shmall=$shmall >> $file
fi
echo "-- Using: new sysctl values"
sysctl -p
fi

# Done
echo "--- Done."

Ctrl-w

Ctrl-x

Make it executable:

chmod 777 pada.sh

Then run it:

./pada.sh

Have not needed to mess with shared memory settings in quite a few versions of ZM. This may actually cause you problems if you use it!

[knight-of-ni]

shmall and shmax were abandoned, in favor of mmap /dev/shm, beginning with zoneminder version 1.25.
Changing these settings today will not affect ZoneMinder at all.

[knight-of-ni]

I have been getting intermittent bursts of errors like this.

ERR [Incomplete frame write: 219264 of 289380 bytes written]

my cat /proc/sys/net/core/wmem_max was also 212992. Last night I took my high res camera to 24-bits and set net.core.wmem_max = 262144 in /etc/sysctl.conf.
I put this also in FRAME_SOCKET_SIZE

(262144 = 256k)

This message is actually the reason why we are getting rid of the frame server in the next release. The frame server is actually slowing things down rather than speeding up the process. The frame server might have been helpful back in the day with older, slower hardware and low resolution cameras but it only seems to cause problems today. You should run with the frame server off (the default).


Also note that, you will get a performance boost if you set all your cameras to 32bit color depth.

[apbb2]

shmall and shmax were abandoned, in favor of mmap /dev/shm, beginning with zoneminder version 1.25.
Changing these settings today will not affect ZoneMinder at all.

Good to know thanks.

It is out of my install items to do & I have gone back to the default settings.

[clintwk]

Is there any chance you can post the settings you used to add your Foscam R2 camera? I would greatly appreciate it!

Thanks in advance!