Running a private net for your cams? Consider installing ntpd to serve them

[jwarfin]

So, I tripped over this issue recently & I thought I'd share info on it.

I recently setup a new 1.30.4 ZM system on Xubuntu 18.04. For various reasons, I chose to setup a private net for the cams. In essence: added a gigabit NIC to the server, established a private subnet for it and hooked the cams up to it via a dedicated POE switch. In my case, cam config of the actual cams could only be done via a Windows app. So, I had to temporarily put the cams on the LAN net to do that. In any case, the cams work great on the private net. But I discovered something after a while...

The cams could not reach a time server and this caused them to hammer the private net with constant retries. The private net is a non-routable subnet, so I knew the cams couldn't sync - I didn't really care about that. But I didn't expect them to NEVER give up retrying on time sync and rapidly flood the NIC with retry packets. Other devices I've used in this kind of scenario will give up on time sync after a few failed attempts and then attempt retrying every few hours of so. Not the case with my cams.

I discovered this situation by running "iftop -n -P" for the private net port on the server. This showed the constant ntp retry packets.

So I setup an ntpd server (actually, chrony) on the ZM server and configured it to allow client time sync from the private cam net. This made the cams happy and eliminated the ntp retry storm on the private net. This reduced load on the NIC and, as a result, reduced the system load factor by about .10. Bonus!

[jwarfin]

Kind of a follow-up to my first post.

It turns out that chronyd will go offline & exit if the network drops. It's a feature, not a bug. The man page hints at the reason behind this. So, it's typical to mitigate this by setting up an hourly cron script to restart chronyd if it's inactive/dead. The ZM setup I'm implementing on is 1.30.4 on Ubuntu 18.04. I implemented the chronyd watchdog this way:

Created /etc/cron.hourly/chrony_watchdog script. Contents:

Code: Select all

#!/bin/sh -e

/usr/local/bin/restart_chrony_if_inactive

Created /usr/local/bin/restart_chrony_if_inactive. Contents:

Code: Select all

#!/bin/bash

CHK=`systemctl is-active chrony | grep -c inactive`

if [ $CHK -eq 1 ]; then
	systemctl restart chrony
	logger -t CHRONY_WATCHDOG "Chrony inactive, restarted" 
fi

exit

Both scripts are owned by root and mode 755. They have to be executable or anacron/cron won't run them.

Once implemented, chronyd will be checked to see if it's running. If not, it will be restarted and the restart will be logged to syslog.

[alabamatoy]

This is useful, thanks much!

[jwarfin]

A further follow-up:

[ FYI: this is related to my work with ZM on Ubuntu 18.04]

Despite setting up the chronyd watchdog script, I'd still see chronyd exit sometimes or not always start at boot. Some googling found this:

https://bugs.launchpad.net/ubuntu/+sour ... ug/1779621

Basically, if the systemd-timesyncd service is also enabled then it effectively collides with chronyd - causing chronyd to exit or just not start. If you read the above launchpad thread, it's unclear if this is an actual bug or just a poorly understood config issue. Installing and enabling chronyd is apparently supposed to disable systemd-timesyncd from starting - but that apparently doesn't always happen.

The bottom line (I believe) is that one should probably have only one time server running on the system. That is, it's not feasible to have chronyd, ntpd and timesyncd all running on the same system and competing for the same ntp port(s). So, to fix the issue, I made sure timesyncd was disabled:

Code: Select all

sudo systemctl disable systemd-timesyncd

In my case, I actually have an upstream ntp server that I rely on for the centralized time reference. So, in my /etc/chrony/chrony.conf, I comment-out the default pool servers and just add a server line for my upstream ntp server:

Code: Select all

# About using servers from the NTP Pool Project in general see (LP: #104525).
# Approved by Ubuntu Technical Board on 2011-02-08.
# See http://www.pool.ntp.org/join.html for more information.
#pool ntp.ubuntu.com        iburst maxsources 4
#pool 0.ubuntu.pool.ntp.org iburst maxsources 1
#pool 1.ubuntu.pool.ntp.org iburst maxsources 1
#pool 2.ubuntu.pool.ntp.org iburst maxsources 2

server 10.2.11.122 trust

To allow the cameras to use chronyd for time sync, I added this to the chrony.conf file:

Code: Select all

allow 10.2.11.215
allow 10.2.21

The 1st line is the IP of the ZM server (where chronyd is running). The 2nd line describes the private net (via the 2nd NIC in the ZM server) the cams are on (eg: cam 1 is 10.2.21.11, cam 2 is 10.2.21.12, ...).