Best Practice/ Optimizing, redux

[Quantum]

Local ZM server in a KVM VM connected to 3 local cameras -- and 3 remote cameras through a WireGuard tunnel. All cameras are fairly high-spec, such as the Dahua PDW5849.

Local cameras work alright, but I got about 1fps when I set up a ZM slave in the remote nanocomputer (ODroid N2+), which I put down to the slowness of the OS SD card in the nano. So I decided to flange the remote RTSP/RTP streams directly through to the local ZM server. At the remote nano using jnettop I see a TCP Tx rate of 4.71Mbps, way below the ping rate of 300Mbps. ('speedtest')

Ok so Debian VLC does not do RTSP/RTP but with SMPlayer I get about 3fps in the remote nano, and local server VM. Not so good, it really should be better. The optical connection pings at 300Mbps so I should think there's plenty of bandwidth.

Viewing ZM console in the Konqueror browser it keeps getting murdered by the OOM daemon... or at least maybe the zmc daemons are, I can't tell, but I have almost no traffic for the remote cameras in the console. I've dedicated 10 vCPUs and 24,576GB of RAM to the VM. (6 hi-rez monitors, 6 lo-rez monitors, the lo-rez looking for motion, and triggering the hi-rez on motion -- if I have it set up correctly)

In the console it shows, oddly, Memory: 55% Swap: 99%.

I don't understand what's going on here? Shouldn't there be much better performance?

[Quantum]

I've upgraded to 1.37.64 and the memory leak seems to have gone away.

But most of my cameras show offline.


I can pull the raw RT[S]P stream into smplayer, and the bitrates are amazingly low, considering the resolutions. Although, the framerate is way below spec, like 1-2fps in SMPlayer. I don't understand this... these cameras should be spraying like a firehose.

235KB/s (2.35Mb/s)


202KB/s (2.02Mb/s)


968KB/s (9.68Mb/s)


1.01MB/s (10.1Mb/s)


(A Byte is 8 bits, with 1 start and 1 stop bit, so 10 total)

The first two streams are on the local LAN and the latter two are coming in on a WireGuard VPN, tested at 300Mb/s. These images were taken on the ZM server KVM VM on Debian.

All screenshots were taken at about the same time of morning while it was pretty dark. Note the amazing night color on two cameras.

I am definitely not overloading the network, and CPUs are about 30%. (green in circle) Memory: 41% Swap: 0%. I don't understand what the problem is.

[burger]

I would try using a conventional network speed tool such as iperf (i think there are others as well) instead of speedtest. ARM boards are notorious for being underpowered compared to x86, so I wouldn't be surprised there. Though you should be able to run (with passthrough and modect on the low res stream) some reasonable framerates. At least with an RPI4. I can't speak for the odroid. I would set them up like so: https://wiki.zoneminder.com/Dummies_Gui ... Zoneminder

If the SD card is slow, you can use an SSD. That's what I did with my RPI4 setup. SD cards are not as slow as they once were though.

VLC should be able to playback RTSP streams. Not sure about RTP.

It would be interesting to see some results of people's speeds with remote cameras. We would need some kind of survey to contrast / compare. Maybe this is an option for the telemetry.

VMs are discouraged here for performance reasons.

What do some iperf or transfer tests from p2p show?

[Quantum]

Thanks for the response. (RTSP is just a control protocol. RTP is the actual video stream)

I've always distrusted these 'speed tests' as I suspect they are based only on ping. Great tip on iperf.

Local to remote cameras server:

Code: Select all

# iperf -c 10.2.3.60 
------------------------------------------------------------
Client connecting to 10.2.3.60, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.2.1.2 port 56260 connected with 10.2.3.60 port 5001 (icwnd/mss/irtt=13/1380/294653)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-22.3117 sec  10.3 MBytes  [color=#FFff00]3.87 Mbits/sec[/color]

# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 10.2.3.60 port 5001 connected with 10.2.1.2 port 56260 (icwnd/mss/irtt=13/1380/287048)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-24.1417 sec  10.3 MBytes  [color=#FFff00]3.57 Mbits/sec[/color]

Remote cameras server to local:

Code: Select all

# iperf -c 10.2.1.2
------------------------------------------------------------
Client connecting to 10.2.1.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.2.3.60 port 41050 connected with 10.2.1.2 port 5001 (icwnd/mss/irtt=12/1300/473101)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-24.1376 sec  9.19 MBytes  [color=#FFff00]3.19 Mbits/sec[/color]

# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 10.2.1.2 port 5001 connected with 10.2.3.60 port 41050 (icwnd/mss/irtt=12/1300/418269)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-150.5493 sec  9.19 MBytes   [color=#FF0000]512 Kbits/sec[/color]

Well that's pretty terrible, considering I have an ostensible 300Mb/s connection.
local ==> WireGuard server VM ==> WireGuard ==> Remote cameras server

I've put in the very fastest SD card in the remote cameras server. Maybe the problem is WireGuard.

Testing directly:

Code: Select all

------------------------------------------------------------
Client connecting to enclave2.dyn.ch, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.2.1.2 port 50266 connected with 50.47.171.90 port 5001 (icwnd/mss/irtt=14/1460/8693)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.1276 sec   102 MBytes  [color=#00FF00]84.6 Mbits/sec[/color]

# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  1] local 50.47.171.90 port 5001 connected with 50.35.183.152 port 50266 (icwnd/mss/irtt=14/1460/8836)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.1186 sec   102 MBytes  [color=#00FF00]84.7 Mbits/sec[/color]

So wow, WireGuard throws a major monkeywrench into the works. I don't know of any newer or faster VPN to use. CPU in the WireGuard server is negligible, and I've just doubled its RAM and iperf got worse!

How are others doing this? Do I have to just set up ZM on the remote server as its own master and have one for home and one for work, and depend on nginx/SSL?

[Andyrh]

Be sure you tune WG. Packet size is very important. I ended up with an MTU of 1420 to get to 750Mb/s.
The VPN header takes some of the packet space.

[burger]

That is good to know about Wireguard. I found this about the MTU: https://superuser.com/questions/1537638 ... termittent

If you have a firewall on both sides that you can configure with iptables of nft, you can
skip using a VPN altogether. Just whitelist the appropriate IP addresses on each side.
Make sure to test the rules are actually working and not allowing in the internet.

VPNs are over promoted, when simple firewall rules will work. People overlook this.
If the client isn't roaming around the internet, you may not need a VPN. At least, it will
be fine with a camera. There is traffic where you should use a VPN, but you can also
put other traffic over the internet without a VPN, and nothing will ever happen. As long
as it's explicitly whitelisted on both sides and blocked to the internet. And it's not a wide
open protocol like telnet or something (though it's highly unlikely the ISPs would be
interested in the traffic).

A static IP here will make it easier. DDNS in iptables can break when the IP changes.
You can sort of get around this by rebooting each day, or scripting some kind of check in, but it's not ideal.

From my understanding once a DDNS is set in iptables, it doesn't do a lookup again
until the firewall is reset or the system reboot. So you could lose connection in the middle of a day if the IP
changes. See: https://askubuntu.com/questions/6714/ip ... date-rules

[Quantum]

DDNS both ends -- no I'm not going to constantly monitor IPs.

I have WG set to 1420 both ends -- terrible throughput. Not CPU/memory/nor disk bound. I'll be looking into nginx proxying to see if it can handle RTSP over SSL. Or maybe reverse SSH tunnels.

Meanwhile ZM has collapsed into a steaming heap, and even after rebuilding the whole VM, "404 not found". What's not found? It's a mystery. Only thing in logs is access.log:

Code: Select all

127.0.0.1 - - [26/Oct/2024:10:03:08 -0700] "GET /favicon.ico HTTP/1.1" 404 125 "http://127.0.0.1/zm/" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"

Under nginx ZM server { with error_log /var/log/nginx/zoneminder_error.log warn; and access_log /var/log/nginx/zoneminder_access.log; ... I get a bonus in nginx's error.log:

Code: Select all

127.0.0.1 - - [26/Oct/2024:10:16:13 -0700] "GET /zm/ HTTP/1.1" 404 125 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"

No sh*t. zoneminder_error.log and zoneminder_access.log are zero-length. The example conf which comes with the Debian package can't work, since it has no server directive.

I can not get the simplest things done, nor to keep working.

Ok sites-enabled/default has to be torn out. Then ZM works. FWIW here are my notes for Debian 12.6, nginx, and ZM 1.37:

# apt install fcgiwrap lsb-release mariadb-server nginx php php-fpm php-mysql gnupg2
# echo "deb https://zmrepo.zoneminder.com/debian/master "`lsb_release -c -s`"/" | sudo tee /etc/apt/sources.list.d/zoneminder.list
# wget -O - https://zmrepo.zoneminder.com/debian/ar ... eyring.gpg | sudo apt-key add -
MAKE SURE it says OK.
# apt update
# apt install -y zoneminder

# systemctl status php8.2-fpm mariadb nginx
# adduser www-data video
# systemctl enable --now zoneminder
# apt remove apache2
# rm -rf /etc/nginx/sites-enabled/default

In /etc/nginx/sites-available/default comment out listen [::]:80 default_server;

My sites-available:

Code: Select all

server {
        server_name localhost;

        root /usr/share/zoneminder/www;

        index index.php;

        access_log /var/log/nginx/zoneminder_access.log;
        error_log /var/log/nginx/zoneminder_error.log warn;

    location /zm/cgi-bin {
        alias /usr/lib/zoneminder/cgi-bin;

        include fastcgi_params;

        fastcgi_param SCRIPT_FILENAME $request_filename;
        fastcgi_param HTTP_PROXY "";

        fastcgi_pass unix:/run/fcgiwrap.socket;
    }

    location /zm/cache {
        alias /var/cache/zoneminder/cache;
    }

    location ~ /zm/api/(css|img|ico) {
        rewrite ^/zm/api(.+)$ /api/app/webroot/$1 break;
        try_files $uri $uri/ =404;
    }

    location /zm {
        alias /usr/share/zoneminder/www;

        try_files $uri $uri/ /index.php?$args =404;

        location /zm/api {
                rewrite ^/zm/api(.+)$ /zm/api/app/webroot/index.php?p=$1 last;
        }

        location ~ \.php$ {
                include fastcgi_params;

                fastcgi_param SCRIPT_FILENAME $request_filename;
                fastcgi_param HTTP_PROXY "";

                fastcgi_index index.php;

                fastcgi_pass unix:/run/php/php-fpm.sock;
        }
    }
}

[eracc]

It is possible to run a "silent" root cron watchdog script when using random IPs with DDNS to check the public IP address and then update firewall rules. I do that with pf on FreeBSD every 30 minutes here with a bash script. One could do it with iptables on Linux as well. It is just a matter of figuring out all the bits, building the script, then setting up a cron job. For dedicated camera streams the check will probably need to be a little faster than 30 minutes.

[Quantum]

I run a freedns script periodically to check for IP changes and update afraid.org upon change. I sure don't want to dynamically mess with my nftables script. (except I do wish that fail2ban could set bans in nftables)

I'm a real estate developer and just don't have time to bit-twiddle. Sorry for my impatience but my attempt to repay is above.

[eracc]

...
Meanwhile ZM has collapsed into a steaming heap, and even after rebuilding the whole VM, "404 not found". What's not found? It's a mystery. Only thing in logs is access.log:

Code: Select all

127.0.0.1 - - [26/Oct/2024:10:03:08 -0700] "GET /favicon.ico HTTP/1.1" 404 125 "http://127.0.0.1/zm/" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"

Under nginx ZM server { with error_log /var/log/nginx/zoneminder_error.log warn; and access_log /var/log/nginx/zoneminder_access.log; ... I get a bonus in nginx's error.log:

Code: Select all

127.0.0.1 - - [26/Oct/2024:10:16:13 -0700] "GET /zm/ HTTP/1.1" 404 125 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"

No sh*t. zoneminder_error.log and zoneminder_access.log are zero-length. The example conf which comes with the Debian package can't work, since it has no server directive.
...

I just noticed your 404 problem there. When I first set up my new ZM install with nginx earlier this week, I was getting 404 messages too. Then I realized nginx already has a default host running on port 80. So I moved the ZM server to listen on port 8080 and that resolved the problem.

Code: Select all

server {
        listen 8080;

        root /usr/local/www/zoneminder;
        index index.php
        gzip off;
...

Since this is my rfc 1918 based home server that is not facing the internet I just use http://ip.ad.dr.ess:8080/zm to access the web interface here in the house. No DNS or DDNS needed.

I am a Unix / Linux Systems Engineer for a large, regional ISP. Bit twiddling is my job.

[Quantum]

That's one way. I just got default out of the way. 127.0.0.1 still shows the default nginx 'site'.

Now ZM isn't getting my RT[S]P feeds. I see the traffic with tcpdump, but ZM just isn't getting it. No video; 2 of 3 cameras red dot, and no image on the third.

I'm too drunk and afraid to try and set up streams from the remotes at my jobsite. I have a remote nanocomputer and PoE switch 'running' 3 cameras on my jobsite 10 miles away which I -must- have access to, but WG can't handle it. Looks like ZM can't even handle my local PoE cameras. Idk what else to do. I am not running windows.

I've been trying and failing with ZM for 13 years. What a loser I am.

[eracc]

You are a Real Estate developer, not a tech geek. No one is thinking you are a loser. Trying to solve your problems over the forum will be a challenge.

See if you have a Linux User Group or FOSS User Group in your area. Join up, and see if you can get a Linux nerd to come assist you for a meal or "free" beverages of choice. A lot of us tech nerds are happy to help people.

[Quantum]

Ok it's a new day. For this KVM VM I've upped CPU cores to 10 and RAM to 24576 and now when I click on monitor 1-2 I get ~15fps. But 3-4 and 5-6 are still red-dot and show offline. I get no stream at all on 5-6 in smplayer, so something must be wrong with that camera.

In Console, no pictures on all. My favorite cracked picture.

It limps along until something gets murdered and I briefly have pictures in Console on 1-4. Then quickly nothing again. ZM 1.37.64

Meanwhile I need a way to securely stream those three remote cameras on my jobsite. WireGuard is clearly not up to it, so now I'm trying to figure out QUIC and MASQUE.

Edit: Oops, Cloudflare is sponsored by the Eff Bee Eye. This looks like a likely candidate though. Why in the World does anyone bother with Go, in the 21st Century, when there's Rust?!

And this looks like a very cool way to bypass the vulns in SSH.

No idea whether I can make either work yet though.

[Quantum]

Compiled my first Rust program. Haven't tried to use it yet.

Checked every aspect of my ZM VM and all is quiet, and ZM still isn't working.

Networking on the cameras interface is running at 13.2 Mb/s with two high-def and two low-def streams... it's a 1GB LAN.


Looks like ZM is the problem. Maybe I should go back to 1.36.

[Quantum]

Ok, ZM v1.36.35 is running just fine so far. Same everything except new mariadb and old ZM. /etc/nginx/sites-available/zoneminder.conf is the same as before. Storage is the same.

There's some problem with 1.37.