SOLVED Apache access denied - intermittent (1.36.31)

[Hillwalker]

I've been using Zoneminder for a few years now (thank you!) and things have mostly gone very well. But now I have a problem with "Forbidden You don't have permission to access this resource." popping up, in ways that don't seem to be consistent and I'm very confused. I recently accepted an upgrade to 1.36.31, so that might be the trigger - though I must own up to have been fiddling a lot with my Apache configuration recently so it is quite possible that I've caused the problem myself.

I have my own domain, lets call it mydomain.com, with various subdomains all pointing at my fixed IP address, with port 80 forwarded to my server. There are a few sites here, and I have Apache Virtual hosts set up. My other sites are working ok, as far as I can see. I'm on Ubuntu 20.04.

If I access zm.mydomain.com/zm then all looks good. I can select various views and options and it works. Then I'll get 403 Forbidden pop up, and there doesn't seem to be any rhyme or reason to it. The error will persist for several page accesses, and then it will work. Even something simple like http://zm.mydomain.com/zm/index.php?view=console will work sometimes and not others.

My zoneminder.conf file is the default, with the addition of

Code: Select all

<VirtualHost *:80>

    ServerName zm.mydomain.com
    ServerAlias www.zm.mydomain.com
    DocumentRoot /usr/share/zoneminder/www

at the beginning (and </VirtualHost> at the end, of course!).

I do have zmninja set up and it mostly works, though I have seen some failures to load images there as well (but of course with other elements to that particular chain that could be unrelated).

Can anyone offer any clues as to what might be going on here?

TIA

[iconnor]

My first thought is timezone difference causing auth to timeout before it should.

[Hillwalker]

Thank you for the thought - I have to admit that timezones hadn't occurred to me as an issue.

If this were the problem, wouldn't I see a demand to re-authenticate? This isn't happening - it just works (eventually, and only for a while). Or is my browser silently doing that for me?

[Hillwalker]

This gets odder and odder the more I play with it. Just going back and forth between the console and http://zm.mydomain.com/zm/index.php?view=watch&mid=1 it'll work ok a few times and then drop into "forbidden" mode.

Or I'll get to the event list and there will be no events visible. Or the events will be there but some of the thumbnails will be missing. Or it'll all look fine but spontaneously change to "forbidden".

(ZmNinja is also pretty unhappy, as one might expect with such issues)

Looking in the Zoneminder Log, I am not seeing an entry per incident, or anything remotely like that. However there are some entries but I've no idea of their significance.

I had a few of these in the early hours of this morning:

11/11/22, 7:27:26 AM UTC web_php 1958065 ERR Socket /run/zm/zms-003768s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit. Please go to http://zoneminder.readthedocs.io/en/lat ... window-etc for more information. includes/functions.php 1880

And, while I was playing with this yesterday, there were quite a few of these:

11/10/22, 3:49:13 PM UTC web_js 1883355 ERR ReferenceError: secsToTime is not defined zm/cache/js_MonitorStream-base-1666048354.js 398

... but I've no idea if these are normal.

[Hillwalker]

OK, well I did say I'd been fiddling with Apache!

For the benefit of anyone else who might fall into the same trap, my problem was caused by the fact that I'd enabled the "evasive" module in Apache, which is intended to avoid DDoS attacks. It works by looking out for a client that demands too many pages too quickly, and returns 403 when the limit is exceeded. This showed up in the apache error log.

You can set the limits, but of course when I'd been thinking about what they ought to be I'd only been thinking about a regular web site.

Zoneminder, it seems, regularly makes tens of separate requests a second - I didn't narrow down the exact threshhold, but setting the limit to hundreds of accesses per second seems to let it work. It also makes the DDoS prevention pretty useless, of course!

If I really want to make my sites robust, I guess I might have to separate them out into different servers which can have their own configuration.

[Magic919]

Have a look at Crowdsec.

[Hillwalker]

Thanks, I will.

[Pennarddog]

This is an old post I know but I’ve got exactly the same issue. My install is out of the box on Turnkey - I’ve not changed my Apache configuration. What exactly did you do in Apache to resolve the Firbidden error ?

[wildfire]

This is an old post I know but I’ve got exactly the same issue. My install is out of the box on Turnkey - I’ve not changed my Apache configuration. What exactly did you do in Apache to resolve the Firbidden error ?

I'm in the same boat, out of box and still getting these issues. Can watch a few recordings, but then it happens after a bit, then locked out of viewing any for a while.

[iconnor]

Sounds like a timezone issue. Make sure your system and ZM timezone match.

[iain]

I've just had this same problem.
A Turnkey install by default has the mod_evasive plug-in enabled, which by default has quite an aggressive default for blocking multiple requests that is triggered by ZoneMinder.

To disable the plug in just delete the following 2 links:
# rm /etc/apache2/mods-enabled/evasive.conf
# rm /etc/apache2/mods-enabled/evasive.load

restart apache:
# systemctl restart apache2

After that the 'Forbidden' errors disappeared for me.

If you ever want to re-enable the mod_evasive plugin again you just need to create new symbolic links to the files in the apache2/mods-available directory.


Another point to note is that the TurnKey image for ZoneMinder also has the fail2ban service enabled by default.
I also decided to disable that (after having problems connecting via SSH to docker image). Which I did using the command:
# systemctl disable fail2ban

[wildfire]

OK, well I did say I'd been fiddling with Apache!

For the benefit of anyone else who might fall into the same trap, my problem was caused by the fact that I'd enabled the "evasive" module in Apache, which is intended to avoid DDoS attacks. It works by looking out for a client that demands too many pages too quickly, and returns 403 when the limit is exceeded. This showed up in the apache error log.

You can set the limits, but of course when I'd been thinking about what they ought to be I'd only been thinking about a regular web site.

Zoneminder, it seems, regularly makes tens of separate requests a second - I didn't narrow down the exact threshhold, but setting the limit to hundreds of accesses per second seems to let it work. It also makes the DDoS prevention pretty useless, of course!

If I really want to make my sites robust, I guess I might have to separate them out into different servers which can have their own configuration.

This was also my issue! Running 'a2dismod evasive' and then 'systemctl restart apache2' fixed it! I never enabled it, must have been default on this fresh install, but have had issues that we're bizzare and unable to pin down. Always 403 errors in random intervals.