CGI auth not working behind reverse proxy

[Rapsey]

Hey all!

First of all thank you for making this amazing piece of software. It's been working out great for me, except for this small issue.

I'm running Zoneminder 1.29.0 on Debian Jessie (8.5), from the repo's (1.29.0+dfsg-1~bpo8+1). It's sitting behind a reverse proxy (for SSL offloading), both sides run nginx (1.10.1) so there's no apache in my setup. For the purpose of debugging this problem I have taken SSL out of the equation though, so right now it's all plain HTTP.

The problem appears to be that nph-zms refuses to accept the auth hash when it comes from the reverse proxy. Below are three scenarios with how the /cgi-bin/nph-zms call appears in the access logs of the zoneminder webserver (so not the reverse proxy).

Without ZM auth, with reverse proxy: everything works. (pages, static assets, live streams, FPS display etc)

Code: Select all

HTTP 200 - /cgi-bin/nph-zms?mode=jpeg&scale=100&maxfps=30&buffer=1000&monitor=1&connkey=90632&rand=1472048641

With ZM auth, without reverse proxy: everything works. (login, pages, static assets, live streams, FPS display etc)

Code: Select all

HTTP 200 - /cgi-bin/nph-zms?mode=jpeg&scale=100&maxfps=30&buffer=1000&monitor=1&auth=dd941637871904efbbdd41d7ebf379f8&connkey=394253&rand=1472045506

With ZM auth and reverse proxy: everything works except live streams. (login, pages, static assets etc)

Code: Select all

HTTP 502 - /cgi-bin/nph-zms?mode=jpeg&scale=100&maxfps=30&buffer=1000&monitor=1&auth=dd941637871904efbbdd41d7ebf379f8&connkey=985236&rand=1472047522

Is there perhaps some kind of IP-based auth hash validation that does not take the X_FORWARDED_FOR headers into account?

Any and all help is greatly appreciated. Thanks for taking the time!

[Soljia]

Can you post your NGINX config that contains your ZM directives? I remember having a problem with this in the past, I think it was a nginx config issue.

Are you running Apache for ZM itselfs? I see you posted about having all NGINX.
Below was my old config when I was running ZM in apache behind nginx.

Code: Select all

server {
        listen 443;
        server_name site.com;

        ssl on;
        ssl_certificate /etc/letsencrypt/live/site.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/site.com/privkey.pem;
        keepalive_timeout       60;
        ssl_session_timeout 10m;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers "HIGH:!aNULL:!MD5 or HIGH:!aNULL:!MD5:!3DES";
        ssl_prefer_server_ciphers on;

        location / {
                proxy_set_header        Host $host;
                proxy_set_header        X-Real-IP $remote_addr;
                proxy_set_header        X-Forwarded-For $remote_addr;
                proxy_no_cache 1;
                proxy_pass      https://zmhost/zm;
        }
}

Otherwise, here is my current nginx config.

Code: Select all

server {
    listen 443;
    server_name site.com;

    ####SSL Section####

    ssl on;
    ssl_certificate /etc/letsencrypt/live/site.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/site.com/privkey.pem;
    keepalive_timeout       60;
    ssl_session_timeout 10m;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers "HIGH:!aNULL:!MD5 or HIGH:!aNULL:!MD5:!3DES";
    ssl_prefer_server_ciphers on;

    root /usr/share/zoneminder/www;
    index index.php;

    ####Location Directives####

    location /api {
        rewrite ^/api(.+)$ /api/index.php?p=$1 last;
    }

    location ~ [^/]\.php(/|$) {
#        fastcgi_split_path_info ^(.+?\.php)(/.*)$;
        if (!-f $request_filename) { return 404; }
        expires epoch;
        include /etc/nginx/fastcgi_params;

        fastcgi_pass unix:/var/run/php5-fpm.sock;
#       fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $request_filename;
    }

    location /cgi-bin {
        gzip off;
        root /usr/lib/zoneminder/;

        include /etc/nginx/fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $request_filename;
        fastcgi_pass unix:/var/run/fcgiwrap.socket;
    }

}

[Rapsey]

I'm dumbfounded. For some reason it just works now, without any changes to the configuration.

The only thing that didn't work properly yet was the live stream FPS display over HTTPS. This was because I proxy pass the traffic from reverse proxy to back-end webserver over HTTP, and ZM uses a few non-relative URL's in AJAX and does not implement proxy protocol detection via HTTP headers. I've solved this by editing index.php and replacing

Code: Select all

if ( isset($_SERVER["HTTPS"]) && $_SERVER["HTTPS"] == 'on' )
{
    $protocol = 'https';
}

with

Code: Select all

if ( isset($_SERVER["HTTPS"]) && $_SERVER["HTTPS"] == 'on' ||
    !empty($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https' ||
    !empty($_SERVER['HTTP_X_FORWARDED_SSL']) && $_SERVER['HTTP_X_FORWARDED_SSL'] == 'on' )
{
    $protocol = 'https';
}

Thank you very much for your nginx configurations though, it's nice to have something to compare to. For the sake of helping out visitors from the future I will also include my nginx configs.

nginx.conf - this is identical on both the reverse proxy and back-end web server.

Code: Select all

user              www-data;
worker_processes  2;

error_log  /var/log/nginx/error.log;
pid        /run/nginx.pid;

events {
  worker_connections  1024;
}

http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;

  # Server name prefixed log format
  log_format combined-vhost '$server_name $remote_addr - $remote_user [$time_local] '
                            '"$request" $status $body_bytes_sent '
                            '"$http_referer" "$http_user_agent"';

  access_log  /var/log/nginx/access.log;

  sendfile     on;
  tcp_nopush   on;
  tcp_nodelay  on;

  keepalive_timeout  65;

  gzip               on;
  gzip_http_version  1.0;
  gzip_comp_level    2;
  gzip_proxied       any;
  gzip_vary          on;
  gzip_min_length    1000;
  gzip_disable       "MSIE [1-6]\.(?!.*SV1)";
  gzip_types         text/plain text/css application/x-javascript text/xml application/xml application/rss+xml application/xml+rss application/atom+xml text/javascript application/javascript application/json text/mathml;

  client_max_body_size           2m;
  server_names_hash_bucket_size  64;
  types_hash_max_size            2048;
  types_hash_bucket_size         64;

  proxy_buffer_size              16k;
  proxy_buffers                  4 32k;
  proxy_busy_buffers_size        64k;
  proxy_temp_file_write_size     64k;

  server_name_in_redirect        off;
  server_tokens                  off;

  include  /etc/nginx/conf.d/*.conf;
  include  /etc/nginx/sites-enabled/*;
}

conf.d/ssl.conf - used only on the reverse proxy.

Code: Select all

ssl_certificate /etc/nginx/conf.d/default_cert.pem;
ssl_certificate_key /etc/nginx/conf.d/default_key.pem;

ssl_dhparam /etc/nginx/conf.d/dhparam.pem;

ssl_session_cache shared:SSL:50m;
ssl_session_timeout 180m;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;

ssl_stapling on;
ssl_stapling_verify on;

proxy.conf - used in reverse proxy vhost below.

Code: Select all

proxy_connect_timeout       30s;
proxy_send_timeout          30s;
proxy_read_timeout          30s;
client_max_body_size        64m;
client_body_buffer_size     128k;
proxy_redirect              off;
proxy_buffering             on;
proxy_buffer_size           16k;
proxy_buffers               4 32k;
proxy_busy_buffers_size     64k;
proxy_temp_file_write_size  64k;
proxy_intercept_errors      on;

reverse proxy vhost

Code: Select all

# Actual vhost
server {
  listen      443 ssl;
  server_name zm.mydomain.be;

  access_log  /var/log/nginx/zm_access.log combined;
  error_log   /var/log/nginx/zm_error.log;

  ssl_certificate /etc/letsencrypt/live/zm.mydomain.be/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/zm.mydomain.be/privkey.pem;
  ssl_trusted_certificate /etc/letsencrypt/live/zm.mydomain.be/fullchain.pem;

  # HSTS - 6 months = 15724800
  # add_header Strict-Transport-Security: max-age=2592000;

  root        /var/empty;

  location / {
    proxy_set_header  X-Forwarded-Host $host;
    proxy_set_header  X-Forwarded-Server $host;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  X-Forwarded-Ssl on;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  Host $host;
    proxy_set_header  X-Real-IP $remote_addr;
    proxy_set_header  X-NginX-Proxy true;
    include           /etc/nginx/proxy.conf;
    proxy_pass        http://zm.back.end;
  }
}

# HTTP rewrites + LetsEncrypt validation
server {
  listen      80;
  server_name zm.mydomain.be zm.mydomain.eu zm.mydomain.net zm.mydomain.org zm.mydomain.com;

  # Allow LetsEncrypt domain validation
  location ~ ^/\.well-known/acme-challenge/ {
    default_type text/plain;
    root /var/lib/letsencrypt;
  }

  location / {
    rewrite ^ https://$host$request_uri? permanent;
  }
}

# HTTPS rewrites
server {
  listen      443 ssl;
  server_name zm.mydomain.eu zm.mydomain.net zm.mydomain.org zm.mydomain.com;

  ssl_certificate /etc/letsencrypt/live/zm.mydomain.be/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/zm.mydomain.be/privkey.pem;
  ssl_trusted_certificate /etc/letsencrypt/live/zm.mydomain.be/fullchain.pem;

  location / {
    rewrite ^ $scheme://zm.mydomain.be$request_uri? permanent;
  }
}

back-end web server vhost

Code: Select all

server {
  listen      80;
  server_name zm.mydomain.be;

  access_log  /var/log/nginx/zm_access.log combined;
  error_log   /var/log/nginx/zm_error.log;

  location / {
    root /usr/share/zoneminder/www;
    index index.php;
  }

  location ~ /.*\.php$ {
    root /usr/share/zoneminder/www;
    try_files $uri =404;

    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass unix:/run/php-fpm-zm.sock;
    fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    include /etc/nginx/fastcgi_params;
    fastcgi_intercept_errors on;
  }

  location /cgi-bin {
    gzip off;
    alias /usr/lib/zoneminder/cgi-bin;

    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $request_filename;
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
  }

  location /api/ {
    alias /usr/share/zoneminder/www/api;
    rewrite ^/api(.+)$ /api/index.php?p=$1 last;
  }
}

Anyway, all my problems are solved now.

[Soljia]

Awesome. The IT gods have shine their light upon you.

[dony71]

I'm dumbfounded. For some reason it just works now, without any changes to the configuration.

The only thing that didn't work properly yet was the live stream FPS display over HTTPS. This was because I proxy pass the traffic from reverse proxy to back-end webserver over HTTP, and ZM uses a few non-relative URL's in AJAX and does not implement proxy protocol detection via HTTP headers. I've solved this by editing index.php and replacing

Code: Select all

if ( isset($_SERVER["HTTPS"]) && $_SERVER["HTTPS"] == 'on' )
{
    $protocol = 'https';
}

with

Code: Select all

if ( isset($_SERVER["HTTPS"]) && $_SERVER["HTTPS"] == 'on' ||
    !empty($_SERVER['HTTP_X_FORWARDED_PROTO']) && $_SERVER['HTTP_X_FORWARDED_PROTO'] == 'https' ||
    !empty($_SERVER['HTTP_X_FORWARDED_SSL']) && $_SERVER['HTTP_X_FORWARDED_SSL'] == 'on' )
{
    $protocol = 'https';
}

Thank you very much for your nginx configurations though, it's nice to have something to compare to. For the sake of helping out visitors from the future I will also include my nginx configs.

nginx.conf - this is identical on both the reverse proxy and back-end web server.

Code: Select all

user              www-data;
worker_processes  2;

error_log  /var/log/nginx/error.log;
pid        /run/nginx.pid;

events {
  worker_connections  1024;
}

http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;

  # Server name prefixed log format
  log_format combined-vhost '$server_name $remote_addr - $remote_user [$time_local] '
                            '"$request" $status $body_bytes_sent '
                            '"$http_referer" "$http_user_agent"';

  access_log  /var/log/nginx/access.log;

  sendfile     on;
  tcp_nopush   on;
  tcp_nodelay  on;

  keepalive_timeout  65;

  gzip               on;
  gzip_http_version  1.0;
  gzip_comp_level    2;
  gzip_proxied       any;
  gzip_vary          on;
  gzip_min_length    1000;
  gzip_disable       "MSIE [1-6]\.(?!.*SV1)";
  gzip_types         text/plain text/css application/x-javascript text/xml application/xml application/rss+xml application/xml+rss application/atom+xml text/javascript application/javascript application/json text/mathml;

  client_max_body_size           2m;
  server_names_hash_bucket_size  64;
  types_hash_max_size            2048;
  types_hash_bucket_size         64;

  proxy_buffer_size              16k;
  proxy_buffers                  4 32k;
  proxy_busy_buffers_size        64k;
  proxy_temp_file_write_size     64k;

  server_name_in_redirect        off;
  server_tokens                  off;

  include  /etc/nginx/conf.d/*.conf;
  include  /etc/nginx/sites-enabled/*;
}

conf.d/ssl.conf - used only on the reverse proxy.

Code: Select all

ssl_certificate /etc/nginx/conf.d/default_cert.pem;
ssl_certificate_key /etc/nginx/conf.d/default_key.pem;

ssl_dhparam /etc/nginx/conf.d/dhparam.pem;

ssl_session_cache shared:SSL:50m;
ssl_session_timeout 180m;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;

ssl_stapling on;
ssl_stapling_verify on;

proxy.conf - used in reverse proxy vhost below.

Code: Select all

proxy_connect_timeout       30s;
proxy_send_timeout          30s;
proxy_read_timeout          30s;
client_max_body_size        64m;
client_body_buffer_size     128k;
proxy_redirect              off;
proxy_buffering             on;
proxy_buffer_size           16k;
proxy_buffers               4 32k;
proxy_busy_buffers_size     64k;
proxy_temp_file_write_size  64k;
proxy_intercept_errors      on;

reverse proxy vhost

Code: Select all

# Actual vhost
server {
  listen      443 ssl;
  server_name zm.mydomain.be;

  access_log  /var/log/nginx/zm_access.log combined;
  error_log   /var/log/nginx/zm_error.log;

  ssl_certificate /etc/letsencrypt/live/zm.mydomain.be/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/zm.mydomain.be/privkey.pem;
  ssl_trusted_certificate /etc/letsencrypt/live/zm.mydomain.be/fullchain.pem;

  # HSTS - 6 months = 15724800
  # add_header Strict-Transport-Security: max-age=2592000;

  root        /var/empty;

  location / {
    proxy_set_header  X-Forwarded-Host $host;
    proxy_set_header  X-Forwarded-Server $host;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  X-Forwarded-Ssl on;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  Host $host;
    proxy_set_header  X-Real-IP $remote_addr;
    proxy_set_header  X-NginX-Proxy true;
    include           /etc/nginx/proxy.conf;
    proxy_pass        http://zm.back.end;
  }
}

# HTTP rewrites + LetsEncrypt validation
server {
  listen      80;
  server_name zm.mydomain.be zm.mydomain.eu zm.mydomain.net zm.mydomain.org zm.mydomain.com;

  # Allow LetsEncrypt domain validation
  location ~ ^/\.well-known/acme-challenge/ {
    default_type text/plain;
    root /var/lib/letsencrypt;
  }

  location / {
    rewrite ^ https://$host$request_uri? permanent;
  }
}

# HTTPS rewrites
server {
  listen      443 ssl;
  server_name zm.mydomain.eu zm.mydomain.net zm.mydomain.org zm.mydomain.com;

  ssl_certificate /etc/letsencrypt/live/zm.mydomain.be/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/zm.mydomain.be/privkey.pem;
  ssl_trusted_certificate /etc/letsencrypt/live/zm.mydomain.be/fullchain.pem;

  location / {
    rewrite ^ $scheme://zm.mydomain.be$request_uri? permanent;
  }
}

back-end web server vhost

Code: Select all

server {
  listen      80;
  server_name zm.mydomain.be;

  access_log  /var/log/nginx/zm_access.log combined;
  error_log   /var/log/nginx/zm_error.log;

  location / {
    root /usr/share/zoneminder/www;
    index index.php;
  }

  location ~ /.*\.php$ {
    root /usr/share/zoneminder/www;
    try_files $uri =404;

    fastcgi_split_path_info ^(.+\.php)(/.+)$;
    fastcgi_pass unix:/run/php-fpm-zm.sock;
    fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    include /etc/nginx/fastcgi_params;
    fastcgi_intercept_errors on;
  }

  location /cgi-bin {
    gzip off;
    alias /usr/lib/zoneminder/cgi-bin;

    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $request_filename;
    fastcgi_pass unix:/var/run/fcgiwrap.socket;
  }

  location /api/ {
    alias /usr/share/zoneminder/www/api;
    rewrite ^/api(.+)$ /api/index.php?p=$1 last;
  }
}

Anyway, all my problems are solved now.

I follow your instruction however get this error
I check under directory /var/run/zm but don't find any /var/run/zm/zms-242095s.sock
instead I see /var/run/zm/zmdc.sock
What could be wrong?

2019/01/07 16:26:53 [error] 29852#29852: *20 FastCGI sent in stderr: "PHP message: ERR [Socket /var/run/zm/zms-242095s.sock does not exist. This file is created by zms, and since it does not exist, either zms did not run, or zms exited early. Please check your zms logs and ensure that CGI is enabled in apache and check that the PATH_ZMS is set correctly. Make sure that ZM is actually recording. If you are trying to view a live stream and the capture process (zmc) is not running then zms will exit.

[dony71]

In fact I found out once in a while exist file /var/run/zm/zms-242095w.sock
but why ERR message try to find file /var/run/zm/zms-242095s.sock ??
Why file name different with w instead of s behind?

[dony71]

can anybody tell me what is correct setting in Options->Paths->PATH_ZMS ?
I put /zoneminder/cgi-bin/nph-zms, but still not working