50+ Camera setup, Struggling here

[tmart1129]

Hi all,
To give a brief rundown of my current setup I have a Proxmox Cluster with Ceph storage, and Zoneminder as a VM. I have 32cores dedicated to it, and 60gb of ram. I have TrueNas on a baremetal server with 57TB of space, with a 10gb link between them. We currently have 45 720p PoE cameras, with plans to possibly add more.

We originally were using Milestone xProtect, but we hated it and was way over priced and license BS, so I've been attempting to get Zoneminder properly setup for a month or two now but I run into numerous issues at every step when attempting to fix something else. We only need the recording footage to look back on any possible issues or security, nothing too official or too fancy, just timestamps on the footage, for just HR confirmation of any events that happened.

I originally installed it as 1.34.25 and got it mostly working but still had some glaring issues like API refusing to work, recording footage is choppy or had large 10second jumps, H264 passthrough makes the footage 1-2 frames per 5 seconds, or montage review would crash and bring us to a white screen when ever trying to load more than 1 camera of footage. Once I finally got the recording working properly, which was by disabling Saving Jpegs on each monitor, then i think thats what caused montage review to bug out but not sure.

So, I upgraded ZM to 1.36.33. This brought a few other issues but fixed the H264 passthrough, unfortunately it doesn't have the timestamps, which I know should be done through the Cameras anyways, but since they're on a VLAN, it was easier for ZM to just do it. Now, encoding produces pretty choppy footage, and has been cutting my footage into 10-20s, or even 1-5s long events when set on Mocord. I then re-enabled Frames Only for Jpegs and I think that might have fixed the montage review issue.

Anyways, apologies for kind of just rambling but I'm at a lost at what I'm doing wrong or what the proper setup should be, I just want to get this done so I can get on with the rest of my job. I'll attach pictures of an example of what my current setup is, if you have any recommendations for any of the issues I've said please let me know before I go any more insane, much appreciated.

[mconfortino]

Hi! Wy have some installations with over 60 cameras each. I think your problems is the buffers allocation. We have 3, 0, 0, 0, 0, 0, 1. And save jpegs is disabled. 20fps set in the cameras. It works quite well this way.

[burger]

I have a setup around 30 cameras. You will need powerful hardware. Are these 32 real cores, or virtual? Also, not to be rude but any API misconfigurations are on your end. The API is stable and works, although ZM does have a learning curve, admittedly. This is probably what you are running into. There are always people on the forum (and most everything has been answered here). General concensus here is that you will have better luck with bare metal as opposed to VMs. I'm not familiar with Proxmox. I wrote a whole bunch on the dummies guide on the wiki for beginners that might help. Such as lowering analysis FPS. Doing motion detection on a high res stream is unadvised.

H264 passthrough shouldn't lower the FPS of the cameras. What model of cameras are these? That can also have an impact. Worst case scenario, you can contact one of the devs and pay them to look at your setup.

You might want to look at wiki.zoneminder.com/MySQL as well, for some scripts to ease settings admin.

[tmart1129]

Hi! Wy have some installations with over 60 cameras each. I think your problems is the buffers allocation. We have 3, 0, 0, 0, 0, 0, 1. And save jpegs is disabled. 20fps set in the cameras. It works quite well this way.

So since this post I've gotten everything running quite smoothly, the only issue left is Montage Review gives a white page. It worked after I wiped the DB of all events ( a lot of the footage was lost anyways during my configurations ), so maybe the buffers would help with that but I'm not sure. Load is down to around 9 on average which is drastically better than it used to be. Also increased RAM allowed by PHP which helped a bit

[tmart1129]

I have a setup around 30 cameras. You will need powerful hardware. Are these 32 real cores, or virtual? Also, not to be rude but any API misconfigurations are on your end. The API is stable and works, although ZM does have a learning curve, admittedly. This is probably what you are running into. There are always people on the forum (and most everything has been answered here). General concensus here is that you will have better luck with bare metal as opposed to VMs. I'm not familiar with Proxmox. I wrote a whole bunch on the dummies guide on the wiki for beginners that might help. Such as lowering analysis FPS. Doing motion detection on a high res stream is unadvised.

H264 passthrough shouldn't lower the FPS of the cameras. What model of cameras are these? That can also have an impact. Worst case scenario, you can contact one of the devs and pay them to look at your setup.

You might want to look at wiki.zoneminder.com/MySQL as well, for some scripts to ease settings admin.

They're virtual cores, passed by proxmox. I followed 3 different guides on the API and none seemed to work properly so not really sure there. Agreed, H264 passthrough shouldn't of lowed the FPS, which is why I upgraded and then it fixed it. Theyre multiple different types of ONVIF cameras. The last issue I have left is the montage review going white/grey screen in firefox. In chrome it gives me an HTTP ERROR 500, which I think might have been fixed by allowing PHP and Apache to have access to more ram in their respective php.ini files.

[burger]

which I think might have been fixed by allowing PHP and Apache to have access to more ram in their respective php.ini files.

Well, definitely check the apache and php logs. Might have to enable them. Between zm, php, and apache logs you will usually find some hint of what's broken. Although I should mention that montage review is a 'newer' feature in zm, so it has not been used as long as the traditional montage view. I haven't used it much yet, personally. I assume you've already dealt with the limitations of 6 cameras in web browsers, and the port management to bypass that. I could take a wild guess and suggest creating groups to lower the number of cameras in montage review, but that's just a guess. I'll try to check montage review and see how it works on my 1.36.33 server.

[tmart1129]

which I think might have been fixed by allowing PHP and Apache to have access to more ram in their respective php.ini files.

Well, definitely check the apache and php logs. Might have to enable them. Between zm, php, and apache logs you will usually find some hint of what's broken. Although I should mention that montage review is a 'newer' feature in zm, so it has not been used as long as the traditional montage view. I haven't used it much yet, personally. I assume you've already dealt with the limitations of 6 cameras in web browsers, and the port management to bypass that. I could take a wild guess and suggest creating groups to lower the number of cameras in montage review, but that's just a guess. I'll try to check montage review and see how it works on my 1.36.33 server.

Yeah, it sure took me a while to get the 6 camera limitation removed, I initially gave up on zoneminder for it because I couldn't get it to work, especially with SSL. But went back at it a while later and finally got it. We do have quite a few groups setup, but before when any more than 1 camera were selected it would give the white page/500 error. Increasing the RAM to both PHP and apache sure seems to have fixed it so far.

If I try and load too many cameras over a longer period of time now it just times out, which is fine and kind of expected I guess, loading a days worth of footage across 12 cameras is quite a heavy task. Is there a way to increase the timeout time though? I saw one in the apache and php ini files, but it didn't seem to do much I don't think.

[iconnor]

I have to assume that these large installs are for businesses. You know I provide support right? I could have you up and running in a day.

I have tools that can help automate applying settings to the cameras, saving tons of time and removing the need for encoding.

[tmart1129]

I have to assume that these large installs are for businesses. You know I provide support right? I could have you up and running in a day.

I have tools that can help automate applying settings to the cameras, saving tons of time and removing the need for encoding.

Yes, for our business. I've seen that you do yes but I've always preferred to do things on my own so I can learn how to fix them when they break. And its more fun and challenging to do on our own than just outsource, but thank you! I think were pretty stable now at the point were at.

Only small issue is looking back at footage times out, because when going to montage review->history then selecting a time, we have to do the Starting time first, and then that submits and goes back to look at days of footage at once across multiple cameras, when we only want footage from 1 hour days ago. Is there a way to prevents zoneminder from auto-submitting the date? Atleast for the starting time, then end time would submit the web page or something similar.

[dougmccrary]

Try selecting a single monitor first, then set the time, then add monitors as needed.
It's better in dev version 1.37, but probably not what you want on a production system.

[tmart1129]

Try selecting a single monitor first, then set the time, then add monitors as needed.
It's better in dev version 1.37, but probably not what you want on a production system.

Yeah I know there's work arounds, but I'm normally not the one using it. Normally HR department or a couple managers, who aren't the most technologically inclined. And its essentially my job to make there's as easy as possible. It'll have to do for now, but it seems like such a simple fix that could be done to just have a 'Go' button to submit the filter settings.

[dougmccrary]

Oooh! Maybe you could get them to fund it?

[geraldbrandt]

Oooh! Maybe you could get them to fund it?

THIS! Funding wanted features will keep ZoneMinder alive.

[iconnor]

Seriously, talk to your people about a support contract. I've got no problem quoting, invoicing, etc. This is a business.

I am very happy to work closely with you to make ZoneMinder do exactly what you need.

[tmart1129]

Seriously, talk to your people about a support contract. I've got no problem quoting, invoicing, etc. This is a business.

I am very happy to work closely with you to make ZoneMinder do exactly what you need.

Might be worth looking into, do you have an rough estimate of price/time? Also, have any recommendations for my Database being extremely large? Specifically the Frames and Stats table were over 20gb EACH. I have about 35indoor cams set as Mocord, and the other 10 are outdoor on Record. I saw BULK_FRAME_INTERVAL option, but that doesn't pertain to mocord, and that would only fix the frames table I assume. Setting them all to Record would also prevent most of that issue, but again, still the Stats table to worry about, unless it would fix the both of them.

Nevermind, I jumped the gun a bit. I found the Record_event_stats option in logging. I guess just any suggestion for the frames table besides changing all to plain ol' record?