Page 1 of 1
[HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster
Posted: Wed Nov 15, 2017 1:58 pm
by vox
Music for brain storming : Best Music Mix 2017 - Best of EDM Remixes Of Popular Songs 2017
This tutorial coming from
[Tuto/HowTo] [GNU/Linux] [EN] Multi Server Zoneminder - fully secure and scalable ZM cluster
This tuto cover ZM 1.30.4 version on Ubuntu-server 16.04.
Warning : the procedure is working, but the tuto need to be test and corrected
Introduction
- This system require minimum 3 machines with Quadcore x64 processor (min i3). Not working with ARM.
This tuto is write for Ubuntu but, normally, it's working too on Debian.
In this thread i explain to you how to creat :
- a Database cluster (mariadb + galera)
- a FileSystem cluster (unified disk space with btrfs JBOD on each machine and unified server space over network with glusterfs)
- a ZoneMinder cluster (multi machine)
To facilitate the setting up we config each server to cumulate all the jobs. But it is not obligatory that every machine hosts all the services.
Re: [HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster
Posted: Wed Nov 15, 2017 2:05 pm
by vox
Install and configure the DataBase Cluster and ZoneMinder
- For secure communication between the node of the DataBase cluster, on one machine install a VPN (example OpenVPN or WireGuard) and connect all your Database Servers/Clients on that VPN. With this method you increase security without so much boring manipulations.
- Connect to admin user (root)
- On Ubuntu / Raspbian
On Debian / Ubuntu Minimal
- Add key and repository for the last software version
- On Ubuntu 16.04
Code: Select all
apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://mariadb.cu.be/repo/10.2/ubuntu xenial main'
add-apt-repository -y ppa:ondrej/apache2
add-apt-repository ppa:iconnor/zoneminder
On Debian 9 (stretch)
Code: Select all
sudo apt-get install software-properties-common dirmngr
apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xF1656F24C74CD1D8
add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://mariadb.mirror.nucleus.be/repo/10.2/debian stretch main'
echo "deb http://www.deb-multimedia.org stretch main non-free" >> /etc/apt/sources.list
For other distro, check this link 
- Install all softwares
Code: Select all
apt-key update
apt-get update
apt-get install -y btrfs-tools glusterfs-server attr glusterfs-client mariadb-server galera-3 mariadb-client zoneminder php-gd php-mysql apache2
glusterfs-client => filesystem client software
mariadb-server galera-3 => database cluster softwares
mariadb-client => database client software
zoneminder php-gd php-mysql apache2 => ZoneMinder and friends
- Lunch following commands
Code: Select all
adduser www-data video
chmod 740 /etc/zm/zm.conf
chown -R www-data:www-data /usr/share/zoneminder/
sudo ln -s /etc/apparmor.d/usr /etc/apparmor.d/disable/.sbin.mysqld
mkdir /usr/share/zoneminder/www/sounds
chown www-data:www-data -R /usr/share/zoneminder/www/sounds
chmod 774 -R /usr/share/zoneminder/www/sounds
a2enmod cgi
a2enmod rewrite
a2enmod http2
sed -ie "/AllowOverride None/c\AllowOverride All" /etc/apache2/apache2.conf
a2enconf zoneminder
a2dismod status # enforce security (disable apache2 server status)
echo "ServerTokens Prod" >> /etc/apache2/apache2.conf # enforce security (disable apache2's serverTokens)
echo "ServerSignature Off" >> /etc/apache2/apache2.conf # enforce security (disable apache2's ServerSignature)
sudo echo "User-agent: *" > /usr/share/zoneminder/www/robot.txt; sudo echo "Disallow: /" >> /usr/share/zoneminder/www/robot.txt; sudo chown www-data:root /usr/share/zoneminder/www/robot.txt # disable google indexing
systemctl enable zoneminder
- Add you timezone in /etc/php/7.0/apache2/php.ini (CTRL+w for search), remove the ; before the line. (for europe past "Europe/Brussels", for USA "America/New_York"
- Edit mariadb the configuration's file /etc/mysql/conf.d/galera.cnf
-
Add the following line after have adapt it (ip, hostname) to your config
Code: Select all
[mysqld]
#mysql settings
sql_mode = NO_ENGINE_SUBSTITUTION
binlog_format=ROW
default-storage-engine=innodb
innodb_autoinc_lock_mode=2
query_cache_size=0
query_cache_type=0
innodb_flush_log_at_trx_commit=0
innodb_buffer_pool_size=256M
bind-address=10.8.1.42 # EDIT ME
#Galera settings
wsrep_provider=/usr/lib/libgalera_smm.so # /usr/lib/libgalera_smm.so OR /usr/lib/galera/libgalera_smm.so
wsrep_cluster_name="DataBaseCluster" # this line need be same on all machine !
wsrep_cluster_address="gcomm://hostnameServer1,hostnameServer2,hostnameServer3" # EDIT ME
#wsrep_cluster_address="gcomm://" # uncomment this line of one machine for first start or when your cluster is break
wsrep_sst_method=rsync
wsrep_on=ON
wsrep_node_address="hostnameServer1" # EDIT ME
wsrep_node_name="hostnameServer1" # EDIT ME
wsrep_cluster_address="gcomm://hostnameServer1,hostnameServer2,hostnameServer3" => add the hostname of all your machine. These hostname need be config in your DNS or in your /etc/hosts files
wsrep_node_address="hostnameServer1" => add the hostname (or the static IP address), don't forget to edit your /etc/hosts for add the same address than the bind-address parameter for the hostname of this server
wsrep_node_name="hostnameServer1" => add the hostname of this machine
- On the "first" node of your cluster, stop mariadb
- On this node edit the file/etc/mysql/conf.d/galera.cnf
-
Comment the line with the list of your nodes "wsrep_cluster_address" and add the following line.
- Creat the cluster
- Restart mariadb on all of your nodes
- Check if it's work
Code: Select all
mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"
- screenshot-2017_08_22-Commade_Galera_Cluster_mariadb_Status_Cluster.png (5.88 KiB) Viewed 7287 times
- If it's work, edit on your "first" node the file /etc/mysql/conf.d/galera.cnf for remove "wsrep_cluster_address="gcomm://" and uncomment your node list.
Code: Select all
wsrep_cluster_address="gcomm://hostnameServer1,hostnameServer2,hostnameServer3"
- Edit zoneminder main configuration's file /etc/zm/zm.conf and adapt the following paramaters
- ZM_DB_HOST=10.8.0.42 # need be same than wsrep_node_address, you can use hostname (with /etc/hosts correspondence) or Static IP address
ZM_DB_PASS="your_Super_Strong_DataBase_Password"
ZM_SERVER_HOST= choose an unique name for this server, need be same than you add in ZM Webui in Options -> Servers
- Reload main services
Code: Select all
service apparmor restart
mysqladmin -uroot -p reload
service mysql restart
service apache2 restart
- Creat your Database user
Code: Select all
mysql -uroot -p -e "grant all on zm.* to 'zmuser'@%.%.%.% identified by 'zmpass';"
- Initialize your database
Code: Select all
mysql -uroot -p < /usr/share/zoneminder/db/zm_create.sql
- Start ZoneMinder and check if it's working.
Re: [HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster
Posted: Wed Nov 15, 2017 2:08 pm
by vox
Unify memory
Explanation :
- First we unified disk on each server via btrfs JBOD without striping. (in this tuto /media/superPartition)
After we use glusterfs for creat a virtual JBOD without stripping over all your servers. (in this tuto /media/ZoneMinder)
On all your Data server
- List your disk
-
Recover the path for each disk we want use for camera's data (example /dev/sdb)
- Creat the btrfs partition on one or more disk (you can add more after if need)
Code: Select all
mkfs.btrfs -d single /dev/sdb /dev/sdc
- screenshot-2017_08_11-Tuto-BTRFS-Creation_JBOD_01.png (65.15 KiB) Viewed 7287 times
If after you wan add one disk, use the following command
Code: Select all
btrfs device add /dev/sdc /media/superPartition/
- Creat the mount point
Code: Select all
mkdir /media/superPartition
mkdir /media/ZoneMinder
- Mount the btrfs partition
Code: Select all
mount /dev/sdb /media/superPartition
- Recover the UID of one of your disk in your btrfs partition
Code: Select all
blkid | grep "btrfs" | grep -E " UUID=\"([0-9a-z]{1,}[\-]{0,}){0,}\""
- Edit /etc/fstab
- Adapt the UUID and add the following lines
Code: Select all
UUID=blablabla-uid-de-votre-disque-blablabla /media/superPartition btrfs defaults,rw,noexec 0 0
127.0.0.1:/ZoneMinder /media/ZoneMinder glusterfs defaults,_netdev,backupvolfile-server=hostnameServer2,noexec 0 0
On one lonely Zoneminder server
- Add all your servers (peers) in your glusterfs cluster (repeat the command for each hostname/server)
Code: Select all
gluster peer probe hostnameServer2
- Creat your JBOD over all your servers
Code: Select all
gluster volume create ZoneMinder transport tcp hostnameServer1:/media/superPartition hostnameServer2:/media/superPartition hostnameServer3:/media/superPartition force
- Start your new network JBOD
- Mount your volume
Code: Select all
mount -t glusterfs 127.0.0.1:/ZoneMinder /media/ZoneMinder
- Move the old zoneminder's data folder to the new destination and give it the good permissions.
Code: Select all
mv /var/cache/zoneminder /var/cache/zoneminderOLD
cp -R -p -v /var/cache/zoneminderOLD/* /media/ZoneMinder
chown www-data:root -R /media/ZoneMinder/*
chmod 770 -R /media/ZoneMinder/*
On each server now you can mount the network JBOD via the next command line, or you can directly reboot.
On all your ZM servers, add a symbolic link (or modify the Path of "events", "images" and "sound" in the WEBUI of Zneminder in "Options" => "Path", it's same result)
Code: Select all
ln -s /media/ZoneMinder /var/cache/zoneminder
By default glusterfs don't encrypt communications between server-server and between client-server, for encrypt communications follow this tuto : [Tuto/HowTo] [GNU/Linux] Gluster and TLS (SSL). This is not required if your communications are going through VPN tunnels.
Re: [HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster
Posted: Wed Nov 15, 2017 3:29 pm
by iconnor
Nice tutorial.
Just a couple of points:
1. I found glusterfs to be impractical for ZM use, due to our use of millions of small jpegs. Storing h264 video should work better here though going forward.
2. My dev branch allows multiple storage areas, so removes the need for a unified storage.
3. This all works fine on arm too.
Re: [HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster
Posted: Wed Nov 15, 2017 3:42 pm
by vox
Thanks
I tried to reduce the size to the maximum but hard hard ^^
iconnor wrote: ↑Wed Nov 15, 2017 3:29 pm
1. I found glusterfs to be impractical for ZM use, due to our use of millions of small jpegs. Storing h264 video should work better here though going forward.
I have try on Localhost (on SSD) it's working fine. But i m not a glusterfs fan (i hate his bugs (
example)), if another one filesystem flashes me, the tutorial will be edited.
Of course, in cluster world glusterfs has a bad reputation with little files but, actually, glusterfs is the simplest I've tested. (but well for french speaker, test all cluster filesystem take amount of time with the doc that is never translated
)
iconnor wrote: ↑Wed Nov 15, 2017 3:29 pm
2. My dev branch allows multiple storage areas, so removes the need for a unified storage.
We can't actually on stable branch (1.30.4) no? (i don't have read any parameter for that)
iconnor wrote: ↑Wed Nov 15, 2017 3:29 pm
3. This all works fine on arm too.
Zm work on ARM (but lag, ARM CPU aren't fastly (
screenshot on Ordroid-XU4 (8 core))).
MariaDB cluster not work on ARM (i have try, the galera dependencies are not satisfied). It's not working too with MySQL Cluster on ARM (i have try too and report the problem on mysql dev forum).