[HowTo] Name code Fort Knox - Creat a fully secure and scallable ZM cluster

[vox]

Music for brain storming : Best Music Mix 2017 - Best of EDM Remixes Of Popular Songs 2017
This tutorial coming from [Tuto/HowTo] [GNU/Linux] [EN] Multi Server Zoneminder - fully secure and scalable ZM cluster
This tuto cover ZM 1.30.4 version on Ubuntu-server 16.04.

Warning : the procedure is working, but the tuto need to be test and corrected

Introduction

• This system require minimum 3 machines with Quadcore x64 processor (min i3). Not working with ARM.
  This tuto is write for Ubuntu but, normally, it's working too on Debian.
  In this thread i explain to you how to creat :
  1. a Database cluster (mariadb + galera)
  2. a FileSystem cluster (unified disk space with btrfs JBOD on each machine and unified server space over network with glusterfs)
  3. a ZoneMinder cluster (multi machine)
  To facilitate the setting up we config each server to cumulate all the jobs. But it is not obligatory that every machine hosts all the services.

[vox]

Install and configure the DataBase Cluster and ZoneMinder

1. For secure communication between the node of the DataBase cluster, on one machine install a VPN (example OpenVPN or WireGuard) and connect all your Database Servers/Clients on that VPN. With this method you increase security without so much boring manipulations.
2. Connect to admin user (root)
   • On Ubuntu / Raspbian

     • Code: Select all

       sudo su

     On Debian / Ubuntu Minimal

     • Code: Select all

       su
       

3. Add key and repository for the last software version
   • On Ubuntu 16.04

     • Code: Select all

       apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xF1656F24C74CD1D8
       add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://mariadb.cu.be/repo/10.2/ubuntu xenial main'
       add-apt-repository -y ppa:ondrej/apache2
       add-apt-repository ppa:iconnor/zoneminder
       

     
     On Debian 9 (stretch)

     • Code: Select all

       sudo apt-get install software-properties-common dirmngr
       apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 0xF1656F24C74CD1D8
       add-apt-repository 'deb [arch=amd64,i386,ppc64el] http://mariadb.mirror.nucleus.be/repo/10.2/debian stretch main'
       echo "deb http://www.deb-multimedia.org stretch main non-free" >> /etc/apt/sources.list
       

     For other distro, check this link
4. Install all softwares

   • Code: Select all

     apt-key update
     apt-get update
     apt-get install -y btrfs-tools glusterfs-server attr glusterfs-client mariadb-server galera-3 mariadb-client zoneminder php-gd php-mysql apache2

     btrfs-tools glusterfs-server attr => filesystem cluster softwares
     glusterfs-client => filesystem client software
     mariadb-server galera-3 => database cluster softwares
     mariadb-client => database client software
     zoneminder php-gd php-mysql apache2 => ZoneMinder and friends
5. Lunch following commands

   • Code: Select all

     adduser www-data video
     chmod 740 /etc/zm/zm.conf
     chown -R www-data:www-data /usr/share/zoneminder/
     sudo ln -s /etc/apparmor.d/usr /etc/apparmor.d/disable/.sbin.mysqld
     mkdir /usr/share/zoneminder/www/sounds
     chown www-data:www-data -R /usr/share/zoneminder/www/sounds
     chmod 774 -R /usr/share/zoneminder/www/sounds
     a2enmod cgi
     a2enmod rewrite
     a2enmod http2
     sed -ie "/AllowOverride None/c\AllowOverride All" /etc/apache2/apache2.conf
     a2enconf zoneminder
     a2dismod status # enforce security (disable apache2 server status)
     echo "ServerTokens Prod" >> /etc/apache2/apache2.conf # enforce security (disable apache2's serverTokens)
     echo "ServerSignature Off" >> /etc/apache2/apache2.conf # enforce security (disable apache2's ServerSignature)
     sudo echo "User-agent: *" > /usr/share/zoneminder/www/robot.txt; sudo echo "Disallow: /" >> /usr/share/zoneminder/www/robot.txt; sudo chown www-data:root /usr/share/zoneminder/www/robot.txt # disable google indexing
     
     systemctl enable zoneminder
     

6. Add you timezone in /etc/php/7.0/apache2/php.ini (CTRL+w for search), remove the ; before the line. (for europe past "Europe/Brussels", for USA "America/New_York"

   • Code: Select all

     nano /etc/php/7.0/apache2/php.ini

7. Edit mariadb the configuration's file /etc/mysql/conf.d/galera.cnf

   • Code: Select all

     nano /etc/mysql/conf.d/galera.cnf

     Add the following line after have adapt it (ip, hostname) to your config

     • Code: Select all

       [mysqld]
       #mysql settings
       sql_mode = NO_ENGINE_SUBSTITUTION
       binlog_format=ROW
       default-storage-engine=innodb
       innodb_autoinc_lock_mode=2
       query_cache_size=0
       query_cache_type=0
       innodb_flush_log_at_trx_commit=0
       innodb_buffer_pool_size=256M
       bind-address=10.8.1.42 # EDIT ME
       
       #Galera settings
       wsrep_provider=/usr/lib/libgalera_smm.so  # /usr/lib/libgalera_smm.so OR /usr/lib/galera/libgalera_smm.so
       wsrep_cluster_name="DataBaseCluster" # this line need be same on all machine !
       wsrep_cluster_address="gcomm://hostnameServer1,hostnameServer2,hostnameServer3" # EDIT ME
       #wsrep_cluster_address="gcomm://" # uncomment this line of one machine for first start or when your cluster is break
       wsrep_sst_method=rsync
       wsrep_on=ON
       wsrep_node_address="hostnameServer1"  # EDIT ME
       wsrep_node_name="hostnameServer1" # EDIT ME
       

       bind-address=10.8.1.42 => add the STATIC IP of this server here
       wsrep_cluster_address="gcomm://hostnameServer1,hostnameServer2,hostnameServer3" => add the hostname of all your machine. These hostname need be config in your DNS or in your /etc/hosts files
       wsrep_node_address="hostnameServer1" => add the hostname (or the static IP address), don't forget to edit your /etc/hosts for add the same address than the bind-address parameter for the hostname of this server
       wsrep_node_name="hostnameServer1" => add the hostname of this machine
8. On the "first" node of your cluster, stop mariadb

   • Code: Select all

     service mysql stop

9. On this node edit the file/etc/mysql/conf.d/galera.cnf

   • Code: Select all

     nano /etc/mysql/conf.d/galera.cnf

     Comment the line with the list of your nodes "wsrep_cluster_address" and add the following line.

     • Code: Select all

       wsrep_cluster_address="gcomm://"

10. Creat the cluster

    • Code: Select all

      sudo /usr/bin/galera_new_cluster

11. Restart mariadb on all of your nodes

    • Code: Select all

      service mysql restart

12. Check if it's work

    • Code: Select all

      mysql -u root -p -e "SHOW STATUS LIKE 'wsrep_cluster_size';"

      • 

        screenshot-2017_08_22-Commade_Galera_Cluster_mariadb_Status_Cluster.png (5.88 KiB) Viewed 7209 times

13. If it's work, edit on your "first" node the file /etc/mysql/conf.d/galera.cnf for remove "wsrep_cluster_address="gcomm://" and uncomment your node list.

    • Code: Select all

      wsrep_cluster_address="gcomm://hostnameServer1,hostnameServer2,hostnameServer3"

14. Edit zoneminder main configuration's file /etc/zm/zm.conf and adapt the following paramaters
    • ZM_DB_HOST=10.8.0.42 # need be same than wsrep_node_address, you can use hostname (with /etc/hosts correspondence) or Static IP address
      ZM_DB_PASS="your_Super_Strong_DataBase_Password"
      ZM_SERVER_HOST= choose an unique name for this server, need be same than you add in ZM Webui in Options -> Servers
15. Reload main services

    • Code: Select all

      service apparmor restart
      mysqladmin -uroot -p reload
      service mysql restart
      service apache2 restart

16. Creat your Database user

    • Code: Select all

      mysql -uroot -p -e "grant all on zm.* to 'zmuser'@%.%.%.% identified by 'zmpass';"

      Replace zmpass by a very strong password.
17. Initialize your database

    • Code: Select all

      mysql -uroot -p < /usr/share/zoneminder/db/zm_create.sql

18. Start ZoneMinder and check if it's working.

    • Code: Select all

      service zoneminder start

[vox]

Unify memory
Explanation :

• First we unified disk on each server via btrfs JBOD without striping. (in this tuto /media/superPartition)
  After we use glusterfs for creat a virtual JBOD without stripping over all your servers. (in this tuto /media/ZoneMinder)

On all your Data server

1. List your disk

   • Code: Select all

     fdisk -l

     Recover the path for each disk we want use for camera's data (example /dev/sdb)
2. Creat the btrfs partition on one or more disk (you can add more after if need)

   • Code: Select all

     mkfs.btrfs -d single /dev/sdb /dev/sdc

     

     screenshot-2017_08_11-Tuto-BTRFS-Creation_JBOD_01.png (65.15 KiB) Viewed 7209 times

     If after you wan add one disk, use the following command

     • Code: Select all

       btrfs device add /dev/sdc /media/superPartition/

       Where /dev/sdc is the path of your new disk. The partition need already be mount.
3. Creat the mount point

   • Code: Select all

     mkdir /media/superPartition
     mkdir /media/ZoneMinder

4. Mount the btrfs partition

   • Code: Select all

     mount /dev/sdb /media/superPartition

5. Recover the UID of one of your disk in your btrfs partition

   • Code: Select all

     blkid | grep "btrfs" | grep -E " UUID=\"([0-9a-z]{1,}[\-]{0,}){0,}\""

6. Edit /etc/fstab

   • Code: Select all

     nano /etc/fstab

7. Adapt the UUID and add the following lines

   • Code: Select all

     UUID=blablabla-uid-de-votre-disque-blablabla /media/superPartition           btrfs   defaults,rw,noexec      0  0
     127.0.0.1:/ZoneMinder /media/ZoneMinder glusterfs defaults,_netdev,backupvolfile-server=hostnameServer2,noexec 0 0

On one lonely Zoneminder server

1. Add all your servers (peers) in your glusterfs cluster (repeat the command for each hostname/server)

   • Code: Select all

     gluster peer probe hostnameServer2

2. Creat your JBOD over all your servers

   • Code: Select all

     gluster volume create ZoneMinder transport tcp hostnameServer1:/media/superPartition hostnameServer2:/media/superPartition hostnameServer3:/media/superPartition force

3. Start your new network JBOD

   • Code: Select all

     gluster volume start NomVolume

4. Mount your volume

   • Code: Select all

     mount -t glusterfs 127.0.0.1:/ZoneMinder /media/ZoneMinder

5. Move the old zoneminder's data folder to the new destination and give it the good permissions.

   • Code: Select all

     mv  /var/cache/zoneminder /var/cache/zoneminderOLD
     cp  -R -p -v /var/cache/zoneminderOLD/* /media/ZoneMinder
     chown www-data:root -R /media/ZoneMinder/*
     chmod 770 -R /media/ZoneMinder/*
     

On each server now you can mount the network JBOD via the next command line, or you can directly reboot.

• Code: Select all

  mount /media/ZoneMinder

On all your ZM servers, add a symbolic link (or modify the Path of "events", "images" and "sound" in the WEBUI of Zneminder in "Options" => "Path", it's same result)

• Code: Select all

  ln -s /media/ZoneMinder /var/cache/zoneminder

By default glusterfs don't encrypt communications between server-server and between client-server, for encrypt communications follow this tuto : [Tuto/HowTo] [GNU/Linux] Gluster and TLS (SSL). This is not required if your communications are going through VPN tunnels.

[iconnor]

Nice tutorial.

Just a couple of points:
1. I found glusterfs to be impractical for ZM use, due to our use of millions of small jpegs. Storing h264 video should work better here though going forward.
2. My dev branch allows multiple storage areas, so removes the need for a unified storage.
3. This all works fine on arm too.

[vox]

Thanks I tried to reduce the size to the maximum but hard hard ^^

1. I found glusterfs to be impractical for ZM use, due to our use of millions of small jpegs. Storing h264 video should work better here though going forward.

I have try on Localhost (on SSD) it's working fine. But i m not a glusterfs fan (i hate his bugs (example)), if another one filesystem flashes me, the tutorial will be edited.
Of course, in cluster world glusterfs has a bad reputation with little files but, actually, glusterfs is the simplest I've tested. (but well for french speaker, test all cluster filesystem take amount of time with the doc that is never translated )

2. My dev branch allows multiple storage areas, so removes the need for a unified storage.

We can't actually on stable branch (1.30.4) no? (i don't have read any parameter for that)

3. This all works fine on arm too.

Zm work on ARM (but lag, ARM CPU aren't fastly (screenshot on Ordroid-XU4 (8 core))).
MariaDB cluster not work on ARM (i have try, the galera dependencies are not satisfied). It's not working too with MySQL Cluster on ARM (i have try too and report the problem on mysql dev forum).