Multi-Server LiveStream using external Hostname

[kulshyn]

Hello,

Recently I tried to experiment with Multi-Server setup. I have a regular dual core PC that I always used for ZM. It is running Ubuntu 14.04 LTS. The setup worked, however I am trying to offload some CPU cycles to other devices since this server is used as media center. So I had Raspberry Pi 2 that I wasn't using and I wanted to see if maybe I can built a micro-processor cluster to do all motion detection. Right now I only have 3 cameras but plan to add 6 more. I realize Raspberry Pi 2 is not powerful enough, but right now I am only at concept testing.

After figuring out how to setup (sharing MySQL and Storage) multi-server I am faced with following problems.
1. When watching LiveStream (click on camera name on dashboard), the element that shows images points to server it is being recorded on, for example
<img id="liveStream" src="http://[SERVER-A]/zm/cgi-bin/nph-zms?mode=jpeg&scale=100&maxfps=5&buffer=1000&monitor=2" alt="CAM01" width="1280" height="720">
This is OK if you are withing your internal network, but accessing ZM from outside will result in blank screen.
2. Same effect is when you watch Montage view. From outside of your network you only see multiple blank squares.
3. Assuming the same issue is with zmNinja. Can't view montage

However, if I go to Events from outside of my network and select one of the past events, I can see that when HTML is rendered it uses relative URL (/zm/cgi-bin/nph-zms?mode=jpeg&scale=100&maxfps=5) which works.
I can also view cameras in 'Montage Review' when using 'flat' skin with Zoneminder Web. I assume that page also uses relative path.

It looks like the issue is an easy fix in forming the html object. I will take a look and try to experiment with the fix in PHP and try to submit answer here. Unless of course this already has work around. I just want to document and make sure next update doesn't wipe out my fixes.

[SteveGilvarry]

The only server with access to live view is the one with the camera, all the event based ones work because the events are on shared storage. Use a VPN is probably the easy solution and more secure.

[knight-of-ni]

See the figure here:
https://zoneminder.readthedocs.io/en/st ... erver.html

Multi-server was designed to work on the local LAN only. It was not intended, and will not work, through a firewall. This is by design.
You will need to do what Steve suggested and create a VPN to make your servers appear on the local network.

[iconnor]

You can use apache reverse proxy to forward the requests internally to the exposed web server. Then you just need to get your dns setup right. It's complex, but works well.

[kulshyn]

VPN is not always available and a hassle to use. Think about scenario of accessing ZM UI on guest PC. Or scenario when you need to view to separate ZM networks, for example you have 2 remote location you are monitoring.

The only server with access to live view is the one with the camera, all the event based ones work because the events are on shared storage. Use a VPN is probably the easy solution and more secure.

[kulshyn]

I saw this diagram when I was setting up the system and it doesn't have "Local Only" anywhere. LAN can always be exposed to external client through FW port or DMZ. The drawing shows client connecting to VideoLAN, that's all.
Since we touch the subject of the figure, what is the point of StorageLAN? It is parallel to VideoLAN and serves no purpose. This figure would make more sense if VideoLAN would be disconnected from Storage Server and Database Server, because when images are being served to clients they will have to travel through StorageLAN back to client through Host to which Client connected.

See the figure here:
https://zoneminder.readthedocs.io/en/st ... erver.html

Multi-server was designed to work on the local LAN only. It was not intended, and will not work, through a firewall. This is by design.
You will need to do what Steve suggested and create a VPN to make your servers appear on the local network.

[kulshyn]

iconnor, will you take my changes when I fix this issue, or I have to go through GitHub? I am not familiar with GitHub, I am more of TFS person.

You can use apache reverse proxy to forward the requests internally to the exposed web server. Then you just need to get your dns setup right. It's complex, but works well.

[kulshyn]

Here is the fix for remote WebUI:

in web/includes/Monitor.php

Code: Select all

53			$streamSrc = ZM_BASE_PROTOCOL.'://'.$Server->Hostname().ZM_PATH_ZMS;
54			} else {
55				$streamSrc = ZM_BASE_URL.ZM_PATH_ZMS;
56			}
57	
58			$args[] = "monitor=".$this->{'Id'};

change to

Code: Select all

53			$streamSrc = ZM_PATH_ZMS; //ZM_BASE_PROTOCOL.'://'.$Server->Hostname().ZM_PATH_ZMS;
54			} else {
55				$streamSrc = ZM_BASE_URL.ZM_PATH_ZMS;
56			}
57	
58			$args[] = "monitor=".$this->{'Id'};

This doesn't fix zmNinja.

[knight-of-ni]

I saw this diagram when I was setting up the system and it doesn't have "Local Only" anywhere.

Do you see a firewall shown in the diagram? I'm sure I don't have to tell you that LAN = local are network.
The diagram clearly shows all hardware tied to the same local networks with no firewall in between any of the servers. This is basic stuff.
If there was anything Internet related, you would have seen a reference to WAN.

Since we touch the subject of the figure, what is the point of StorageLAN? It is parallel to VideoLAN and serves no purpose.

Keeping the storage network activity off your production network is key to managing high traffic load. There is no point in mashing all the broadcast traffic onto the same network segment, when it can be so easily split apart. Enable jumbo frames on the Storage LAN for additional performance. Jumbo frames are typically not possible on the video LAN due to incompatible devices on that network. Take a tour of an enterprise network sometime and you'll see how it is done.

Don't build a storage lan if you don't want to. The storage lan is intended to show how to set things up for maximum performance in a large network.

This figure would make more sense if VideoLAN would be disconnected from Storage Server and Database Server,

You aren't looking at the figure closely enough. The diagram does not indicate normal traffic flowing through those links. Those are labelled management links, shown purely for convenient access to manage the storage and database servers. With jumbo frames enabled on the Storage LAN, you can't just plug any old device into the storage lan and get access.

Feel free to reply saying how you wouldn't do it that way, but the fact is I have been doing this for a long time.

Code: Select all

53			$streamSrc = ZM_PATH_ZMS; //ZM_BASE_PROTOCOL.'://'.$Server->Hostname().ZM_PATH_ZMS;
54			} else {
55				$streamSrc = ZM_BASE_URL.ZM_PATH_ZMS;
56			}
57	
58			$args[] = "monitor=".$this->{'Id'};

This proposed change won't work. The url has to be absolute because the stream source of some of your cameras are going to point to a server other than the one you are browsing from. If you have three servers A,B,C and you are browsing from server A, the streams for cameras mapped to server A can be relative, but the streams for cameras mapped to server B & C can't be. That is why $Server->Hostname() is part of $streamSrc.

[kulshyn]

Code: Select all

53			$streamSrc = ZM_PATH_ZMS; //ZM_BASE_PROTOCOL.'://'.$Server->Hostname().ZM_PATH_ZMS;
54			} else {
55				$streamSrc = ZM_BASE_URL.ZM_PATH_ZMS;
56			}
57	
58			$args[] = "monitor=".$this->{'Id'};

This proposed change won't work. The url has to be absolute because the stream source of some of your cameras are going to point to a server other than the one you are browsing from. If you have three servers A,B,C and you are browsing from server A, the streams for cameras mapped to server A can be relative, but the streams for cameras mapped to server B & C can't be. That is why $Server->Hostname() is part of $streamSrc.

It does work for me. I tested from remote location connecting to my home system of Multi-Server setup. Also, as I said before, event view is already doing relative path (that's not to mention that else statement of code above also generates relative path since ZM_BASE_URL is empty). But take a look at the code in web/includes/Event.php

Code: Select all

121 public function getStreamSrc( $args, $querySep='&' ) {
122	    return ZM_BASE_URL.'/index.php?view=view_video&eid='.$this->{'Id'};
…	
146	      $streamSrc .= "?".join( $querySep, $args );
147	    }
148	

and web/index.php

Code: Select all

67	// Use relative URL's instead
68	define( "ZM_BASE_URL", "" );

this created a relative URL. Most browsers (if not all) will populate the beginning with protocol and host of referrer. If need to make that all browser can handle relative path, one can use $HTTP_SERVER_VARS to pull original server url.

Do you see a firewall shown in the diagram? I'm sure I don't have to tell you that LAN = local are network.
The diagram clearly shows all hardware tied to the same local networks with no firewall in between any of the servers. This is basic stuff.
If there was anything Internet related, you would have seen a reference to WAN.

No, but I see Monitor from Anywhere - https://zoneminder.com/features/
I do have hard time finding any fine print that says "except when you are in Multi-Server mode". And Multi-Server setup instructions don't have notes on "Local Only".

[knight-of-ni]

If this appears to be working for you then double check your camera->server mappings. Also check to make sure your zm* processes for each camera are running on the server you think they should be running on (this takes a bit of experimentation since the process table does not directly associate each process to a camera).

When remotely browsing to server #1, open a live view from a camera you are certain is mapped to server #2 (and only server #2).
If multiserver is configured correctly, the streamsource will point to server #2, which will get blocked by your firewall.

If the stream source doesn't point to server #2, then you are not running multiserver. You are instead, running in a mode which was always possible. That mode however, is essentially dual streaming one camera to two servers. Each server is using resources to process the same stream, rather than just one server.

In case you didn't know, you can check the stream source url by hovering over the live stream, right click, and copy to your clipboard.

ZoneMinder has a lot of feature built into it. Some features are mutually exclusive from one another. You have run into a pair of features which currently do not work together. Multiserver is a brand new feature and the first iteration was designed exclusively for a LAN. I'm sure in the future we will officially address viewing through a firewall, but for now you've either got to vpn or set up a reverse proxy like iconnor suggested.

[kulshyn]

If this appears to be working for you then double check your camera->server mappings. Also check to make sure your zm* processes for each camera are running on the server you think they should be running on (this takes a bit of experimentation since the process table does not directly associate each process to a camera).

When remotely browsing to server #1, open a live view from a camera you are certain is mapped to server #2 (and only server #2).
If multiserver is configured correctly, the streamsource will point to server #2, which will get blocked by your firewall.

If the stream source doesn't point to server #2, then you are not running multiserver. You are instead, running in a mode which was always possible. That mode however, is essentially dual streaming one camera to two servers. Each server is using resources to process the same stream, rather than just one server.

In case you didn't know, you can check the stream source url by hovering over the live stream, right click, and copy to your clipboard.

ZoneMinder has a lot of feature built into it. Some features are mutually exclusive from one another. You have run into a pair of features which currently do not work together. Multiserver is a brand new feature and the first iteration was designed exclusively for a LAN. I'm sure in the future we will officially address viewing through a firewall, but for now you've either got to vpn or set up a reverse proxy like iconnor suggested.

I understand. I appreciate developers that have spend their time developing ZoneMinder. I think it is great application that can find many uses.

So here is what my current setup is. It is expiremental, since I am trying to see if ODROID cluster setup is a viable solution.

I have Ubuntu 14.04 LTS with ZM 1.30 on regular Dual Core box. Same box runs MySQL and hosts all the images. This is a server I used to record all cameras. Lets call it SERVER-LNX

I've setup Ubuntu 16.04 Mate on Raspberry Pi 2 with ZM 1.30 yesterday, I know it is not powerful enough, but this is just proof of concept. I configured my RPI (Raspberry Pi) to use my original server MySQL. In addition I mounted folder with all events and images and symlink /var/cache folders to them. Lets call this one SERVER-RPI

On SERVER-LNX, I modified zm.conf to include Hostname of the server. I did the same thing on SERVER-RPI but with its own hostname. I logged into to UI and added this to hostnames to Options->Servers. After that I went to each camera configuration and assigned them to either two of the servers.

I can see all events being recorded correctly. Cameras are green on UI. Livestream and Montage works.

Now, if I log in from outside it behaves differntly. Let's say I have external host name zm.myhome.com and I use port 780 as my forwarding port to SERVER-LNX port 80. I type in http://zm.myhome.com:780/zm in the browser and get to dashboard. I can view any event recorded but not livestream. Inspecting (right click Inspect in Chrome) object gives me URL that gave in my first post of this thread.
<img id="liveStream" src="http://SERVER-LNX/zm/cgi-bin/nph-zms?mo ... &monitor=2" alt="CAM01" width="1280" height="720">
Obviously, it doesn't work since it can't access my home. The change I proposed, changes URL to http://zm.myhome.com:780/zm/cgi-bin/nph-zms?...

So my design, is not exactly what figure explains. However, it works. But even if I am in Local LAN, open UI from SERVER-LNX, and then open camera being processed SERVER-RPI, I will be given url with SERVER-RPI in it. The problem with it, at least for my setup, SERVER-RPI will be pulling images through network share and serving back to me, which is not efficient in my case and would be more efficient to reference those images directly from SERVER-LNX. I understand that this is an issue with my design and if I had separate server for storage it wouldn't matter. But then, if you do have separate storage server, why bother modifying URL with camera specific server hostname. Why not just return URL with hostname of UI server you are currently accessing.

Not sure if I was able to lay out my thought process in form that everyone will understand. I just think it is irrelevant which server recorded camera, it only matters which UI server you are connected to.

[asker]

as far as zmNinja goes, are you on 1.30 of ZM?
zmNinja seems to work fine as long as you are using 1.30. I use iconnor's multi-server/multi-storage system as a testbed.

[kulshyn]

zmNinja montage doesn't work. I can see API calls to alarm API in apache2 accesslog but no calls to stream. I also see a lot of json envelopes with mention of my server hostname in zmNinja debug logs.
Event view works in zmNinja.

[asker]

can you email debug logs to me via zmNinja - for event view and montage view? I'd like to see why. It works for me using iconnor's system.