Page 1 of 2
Lost all network cameras
Posted: Sat Feb 20, 2016 8:58 am
by rip
So, I can't get network cameras to stream or capture after going from 1.28.1 to 1.29.
UBUNTU 14.04 server using PPA iconnor:zoneminder (not master).
I've spent DAYS on this, and tonight what I did was, stop my processes, rename my original database [so I have a reference for every single setting zm renamed to zm_original and zm dropped after purge], removed the PPA, deleted all my events, etc, purged zoneminder and did a pretty near clean install. Even removed the PPA and re-added it just to follow the steps close as possible.
I did not change the ScriptAlias, but did verify that that path really exists and nph-zms is where I expect it to be.
apache2.conf is nearly bone stock (simple entry for phpmyadmin added after this started so I could easily view the db for sanity checking)
I was careful with all the steps in the 1.29 the easy way doc, explicitly following it and pretending I know nothing so I didn't second guess it on the reinstall. a2enmods and a2enconf, etc., all done.
I did have an early issue with socks files (/run/zm) hung up and stopped apache2 and zoneminder and nuked those manually and restarted. Quieter logs but same result. The backend is processing [capturing at a few FPS and blue check signal background]. 858 card (PV-183) based cams are OK and stream and capture.
I suspect something, somewhere in Apache is at blame, since purging zm and rebuilding gave the same result. So please, if someone can point out the obvious and make me look like an idiot, that'd be great. I'm generally pretty good at this kind of thing, but kind of burned out on it after a few nights hacking on it. Been meaning to upgrade off 14.04 but rather wait for 16.04 LTS so don't want to nuke it just yet.
This is a headless server for what it's worth. No desktop or monitor to browse localhost.
PATH_ZMS is /zm/cgi-bin/nph-zms
/zm/api is all green
Code: Select all
getVersion.json
{
"version": "1.29.0",
"apiversion": "1.0"
}
Code: Select all
Clean start apache2 error.log:
[Fri Feb 19 23:54:53.114751 2016] [mpm_prefork:notice] [pid 1736] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.14 configured -- resuming normal operations
[Fri Feb 19 23:54:53.114806 2016] [core:notice] [pid 1736] AH00094: Command line: '/usr/sbin/apache2'
Code: Select all
zmc_m3 [Network Camera, can view in browser and worked in 1.28.1, etc.]
016-02-20 00:01:51.527192 zmc_m3 1642 ERR Unable to get response, disconnecting zm_remote_camera_http.cpp 1141
2016-02-20 00:01:51.472337 zmc_m3 1642 ERR Unable to read content zm_remote_camera_http.cpp 1040
2016-02-20 00:01:51.082095 zmc_m3 1642 ERR Unable to get response, disconnecting zm_remote_camera_http.cpp 1141
2016-02-20 00:01:51.033967 zmc_m3 1642 ERR Unable to read content zm_remote_camera_http.cpp 1040
2016-02-20 00:01:50.647195 zmc_m3 1642 ERR Unable to get response, disconnecting zm_remote_camera_http.cpp 1141
2016-02-20 00:01:50.595472 zmc_m3 1642 ERR Unable to read content zm_remote_camera_http.cpp 1040
2016-02-20 00:01:50.303020 zmc_m3 1642 ERR Unable to get response, disconnecting zm_remote_camera_http.cpp 1141
2016-02-20 00:01:50.157137 zmc_m3 1642 ERR Unable to read content zm_remote_camera_http.cpp 1040
zma_m3 is quiet since the last reboot. Again, m3 is the first network camera.
Code: Select all
syslog mirrors zmc_m3:
Feb 20 00:46:35 server1 zmc[1642]: ERR [Unable to get response, disconnecting]
Feb 20 00:46:35 server1 zmc[1642]: ERR [Unable to read content]
Feb 20 00:46:35 server1 zmc[1642]: ERR [Unable to get response, disconnecting]
Feb 20 00:46:36 server1 zmc[1642]: ERR [Unable to read content]
Feb 20 00:46:36 server1 zmc[1642]: ERR [Unable to get response, disconnecting]
Feb 20 00:46:36 server1 zmc[1642]: ERR [Unable to read content]
Feb 20 00:46:36 server1 zmc[1642]: ERR [Unable to get response, disconnecting]
Feb 20 00:46:37 server1 zmc[1642]: ERR [Unable to read content]
Feb 20 00:46:37 server1 zmc[1642]: ERR [Unable to get response, disconnecting]
dmesg is quiet
mysqld/error.log is quiet
thanks!
Re: Lost all network cameras
Posted: Sat Feb 20, 2016 1:09 pm
by bbunge
You said that you upgraded but inferred that you followed the "Easy Way" instructions? Those are for install.
The upgrade instructions are located here:
https://wiki.zoneminder.com/Common_Issu ... PA_version
May make a difference..
There is also something at the end you need to do which some folks overlook:
Open Zoneminder. Click on Options - Paths
Change PATH_ZMS to /zm/cgi-bin/nph-zms
Re: Lost all network cameras
Posted: Sat Feb 20, 2016 5:42 pm
by rip
Thanks for the reply.
I originally did upgrade and followed the upgrade instructions. Then spent 2 days trying to get IP cams to work. I read tons of stuff all over the net, and made sure the db looked normal, etc. I really did try hard and usually good at hammering these things out.
So I couldn't get it ironed out. OK, no big deal, lets stop the processes, purge zoneminder, drop the zm database, removed the PPA [thought I might of added master on accident, nope, it was good, but still removed it], rm -Rf * all my events and images, etc., made sure .conf files were gone (zoneminder.conf for apache2, zm.conf.) I did allow the timezone in php.ini to remain across tries.
Then ran through the steps of a clean install, which was find with a couple expected bumps (like, a2enmod cgi is already enabled, etc. No harm running the command again.)
Path-zms was one of the first things I listed in my post on purpose.
Re: Lost all network cameras
Posted: Sun Feb 21, 2016 1:23 am
by bbunge
Hmm... I ran the install procedure (from the WIKI) on a fresh Ubuntu 14.04 server install this afternoon and everything worked as it was supposed to work. I did the install from another PC and just copied and pasted the commands into the Putty terminal. You were getting a response from the API so that works. Usually if you do not get video the CGI was not enabled or the PATH_ZMS is wrong.
Could be some permissions: Set permissions of /etc/zm/zm.conf to root:www-data 740
chmod 740 /etc/zm/zm.conf
chown root:www-data /etc/zm/zm.conf
Re: Lost all network cameras
Posted: Sun Feb 21, 2016 11:51 pm
by rip
It's not that either. Still a complete mystery to me. I haven't had time to mess it with much. probably will dump the whole things and try again.
Re: Lost all network cameras
Posted: Tue Feb 23, 2016 1:43 pm
by iconnor
Those error logs show a failure to grab images... you will need to crnak up the debug logging options to at least 3 and see what it logs.
Re: Lost all network cameras
Posted: Sun Feb 28, 2016 7:41 am
by rip
iconnor wrote:Those error logs show a failure to grab images... you will need to crnak up the debug logging options to at least 3 and see what it logs.
Thanks for the reply Isaac. This has be stumped. 12.04 with 1.28.1 worked.
I put the debug at 4 in the snippets below.
I did a complete reload, multiple times, now using 15.10 (from fresh ISO download) and the 15.04/15.10 "the easy way" instructions.
You'll notice that HTTP 1.0 vs 1.1 in the log below, but I've tried both ways. I also thought maybe SHM was running out of memory but it's OK.
I've made sure apparmor was off, just in case.
ZM_PATH is correct at: /zm/cgi-bin/nph-zms
But it's apparently not making Sockets right. Local analog cameras on a 858 card are fine but IP cam is not.
Code: Select all
rip@server1:/run/zm$ ls -la
total 4
drwxr-xr-x 2 www-data www-data 80 Feb 27 22:11 .
drwxr-xr-x 21 root root 660 Feb 27 22:10 ..
srwxr-xr-x 1 www-data www-data 0 Feb 27 22:11 zmdc.sock
-rw-r--r-- 1 www-data www-data 4 Feb 27 22:11 zm.pid
Apache error log:
Code: Select all
Sat Feb 27 22:04:12.924089 2016] [:error] [pid 1055] [client 10.168.1.46:54554] ERR [socket_sendto( /var/run/zm/zms-407307s.sock ) failed: No such file or directory], referer: http://10.168.1.151/zm/?view=watch&mid=1
It looks like it's (trying to anyway) pumping data:
Code: Select all
02/27/16 23:16:27.186517 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/124 [Connected to host, socket = 6]
02/27/16 23:16:27.186548 zmc_m1[1607].DB2-zm_remote_camera_http.cpp/138 [Sending request: GET /image?res=full&x0=0&y0=0&x1=2048&y1=1536&quality=6&doublescan=0 HTTP/1.1
User-Agent: ZoneMinder/1.29.0
Host: 10.168.1.163
]
02/27/16 23:16:27.186577 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/147 [Request sent]
02/27/16 23:16:27.189408 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/207 [Just getting 15]
02/27/16 23:16:27.189438 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/210 [Expecting 15 bytes]
02/27/16 23:16:27.189457 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/232 [Read 15 bytes]
02/27/16 23:16:27.189468 zmc_m1[1607].DB4-zm_remote_camera_http.cpp/238 [HTTP/1.0 200 OK]
02/27/16 23:16:27.189478 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/862 [Unable to extract entire header from stream, continuing]
02/27/16 23:16:27.193178 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/207 [Just getting 623]
02/27/16 23:16:27.193208 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/210 [Expecting 623 bytes]
02/27/16 23:16:27.193227 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/232 [Read 623 bytes]
02/27/16 23:16:27.193238 zmc_m1[1607].DB4-zm_remote_camera_http.cpp/238 [HTTP/1.0 200 OK
Content-Type: image/jpeg
▒▒▒▒]
02/27/16 23:16:27.193255 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/773 [Got status '200' (OK), http version 1.0]
02/27/16 23:16:27.193267 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/814 [Got content type 'image/jpeg']
02/27/16 23:16:27.193608 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/207 [Just getting 1460]
02/27/16 23:16:27.193622 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/210 [Expecting 1460 bytes]
02/27/16 23:16:27.193635 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/232 [Read 1460 bytes]
02/27/16 23:16:27.193645 zmc_m1[1607].DB4-zm_remote_camera_http.cpp/238 [▒▒▒▒]
02/27/16 23:16:27.194097 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/207 [Just getting 1460]
I've run zm for a few years on usually ubuntu distros, so like to think I'm a little literate about it. Been using linux since beta 16 when Linus was still writing it. I'm not an apache expert though, I make equipment for satellite uplinks.
Re: Lost all network cameras
Posted: Sun Feb 28, 2016 4:36 pm
by bbunge
What are the cameras you are using (I looked through the thread and could not see that you mentioned them)? The errors you are seeing tell me that you have a camera setting not just right. Ubuntu 15.10 should run like a champ! You may want to do teh setup with the default ZM 1.28.1 get your cameras working then upgrade.
bb
Re: Lost all network cameras
Posted: Sun Feb 28, 2016 5:22 pm
by rip
The one I'm testing against is a 3mp Arecont 3155DN. I actually exported my camera settings table before all this so I didn't loose my strings, so I know exactly how they were set up. (arecont options can make for hairy path strings.)
Using that info, I can pull images and streams from VLC on my windows machine. I can ping the camera from the ZM machine (Its headless 6 core with 16gb ram so plenty of horsepower.)
I spent a lot of yesterday trying different camera settings, thinking maybe something in zm changed breaking how it was.
The way I was doing it was remote and grabbing individual full frame images (2048x1536) with ZM, as this game me more control over the image than streaming. I even went to the camera itself and took apart the user interface to look at the static and streaming URLs (wrapped in a iframe in the UI) to compare and use try in ZM.
It's probably something stupid and I've been looking at it so long I can't see it. :/ Hate it when that happens.
Re: Lost all network cameras
Posted: Mon Feb 29, 2016 1:51 am
by bbunge
Did you upgrade your database and permissions?
Change permissions in your "ZM database to allow the creation of a table
mysql -uroot -p -e "grant all on zm.* to 'zmuser'@localhost identified by 'zmpass';"
Run the database upgrade
/usr/bin/zmupdate.pl
Re: Lost all network cameras
Posted: Mon Feb 29, 2016 6:07 am
by rip
Ummm, yeah....I formatted the entire system twice and installed from the ground up on 15.10 since multiple tries on 14.04 were not working.
I did do the DB upgrade on the previous tries before wiping it out, and installing (new partition and everything....even tried a couple ways because I was grasping at anything, like possibly LVM was somehow tripping up permissions. Nope.)
Everything works except IP cams.
Re: Lost all network cameras
Posted: Mon Feb 29, 2016 12:04 pm
by mcouture
I had 1.29.0 running for awhile now on Unbuntu and I run the Software Updater and it showed a bunch of updates available, including ZoneMinder.
I ran the updater and now all my cameras do not connect (all wifi/network cams)
Re: Lost all network cameras
Posted: Mon Feb 29, 2016 12:58 pm
by mcouture
I had to run the "zmupdate.pl -f" command.
Re: Lost all network cameras
Posted: Mon Feb 29, 2016 7:56 pm
by bbunge
From the WIKI...
Guides for Ubuntu Desktop
Installing Zoneminder on a desktop system can have its benefits. It can also pose problems!
If you use one of the desktops provided with Ubuntu (Gnome, Unity, KDE, et. al.) use the corresponding Ubuntu server install/upgrade instructions. You may need to install tasksel to make it easier to install LAMP as most of the Ubuntu instructions start with a system set up with LAMP. It is recommended to install Zoneminder from a terminal. Using a software/package manager can get you in trouble!
If you are using a distro such as Mint, find out which Ubuntu system your version is based on and use the corresponding Ubuntu server install/upgrade. And do the procedure from a terminal!
Oh, did I fail to mention it is easy to copy the command instructions from the instructions on the WIKI and paste them into the terminal window? It is a great way to avoid typos!!!
Repeat: Using a software/package manager can get you in trouble!
Re: Lost all network cameras
Posted: Wed Mar 02, 2016 5:38 am
by rip
This is 15.10 x64 server, headless, no gui. Multiple fresh installs, configured via ssh as root.
Something in the sockets broke, and it's none of the usual suspects.
PATH_ZMS is /zm/cgi-bin/nph-zms before we go over that again. Going to IP/zm/cgi-bin/nph-zms does not give a 404, just blank
Some of the below is actually trying to grab /img.jpg from the cam, which is a static jpg no stream to error. It works in a browser on my network (windows and linux machines, but the WEB server can't pull it.
Code: Select all
root@server1:/var/log# cat /var/log/syslog |grep .sock
Mar 1 12:51:16 server1 web_php[9546]: ERR [socket_sendto( /var/run/zm/zms-997856s.sock ) failed: No such file or directory]
Mar 1 15:58:27 server1 web_php[10351]: ERR [socket_sendto( /var/run/zm/zms-219461s.sock ) failed: No such file or directory]
Mar 1 15:58:28 server1 web_php[10351]: ERR [getCmdResponse stream error: socket_sendto( /var/run/zm/zms-219461s.sock ) failed: No such file or directory - checkStreamForErrors()]
Mar 1 20:36:44 server1 web_php[10171]: ERR [socket_sendto( /var/run/zm/zms-253299s.sock ) failed: No such file or directory]
Mar 1 20:36:44 server1 web_php[10171]: ERR [getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-253299s.sock ) failed: No such file or directory - checkStreamForErrors()]
Mar 1 20:37:27 server1 web_php[10351]: ERR [socket_sendto( /var/run/zm/zms-787429s.sock ) failed: No such file or directory]
Mar 1 20:37:27 server1 web_php[10351]: ERR [getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-787429s.sock ) failed: No such file or directory - checkStreamForErrors()]
Mar 1 20:41:34 server1 web_php[10564]: ERR [socket_sendto( /var/run/zm/zms-815036s.sock ) failed: No such file or directory]
Mar 1 20:41:34 server1 web_php[10564]: ERR [getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-815036s.sock ) failed: No such file or directory - checkStreamForErrors()]
Mar 1 20:43:29 server1 web_php[10169]: ERR [socket_sendto( /var/run/zm/zms-684125s.sock ) failed: No such file or directory]
Mar 1 20:43:29 server1 web_php[10169]: ERR [getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-684125s.sock ) failed: No such file or directory - checkStreamForErrors()]
root@server1:/var/log#
Code: Select all
root@server1:/var/run/zm# ls -la
total 4
drwxr-xr-x 2 www-data www-data 80 Mar 1 20:43 .
drwxr-xr-x 22 root root 740 Mar 1 21:17 ..
srwxr-xr-x 1 www-data www-data 0 Feb 28 15:44 zmdc.sock
-rw-r--r-- 1 www-data www-data 4 Feb 28 15:44 zm.pid
root@server1:/var/run/zm#
But I can WGET the image on the server:
Code: Select all
root@server1:~# wget http://10.168.1.163/img.jpg
--2016-03-01 21:34:39-- http://10.168.1.163/img.jpg
Connecting to 10.168.1.163:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [image/jpeg]
Saving to: ‘img.jpg’
img.jpg [ <=> ] 210.17K --.-KB/s in 0.05s
2016-03-01 21:34:39 (4.32 MB/s) - ‘img.jpg’ saved [215213]
root@server1:~# ls -la *.jpg
-rw-r--r-- 1 root root 215213 Mar 1 21:34 img.jpg
root@server1:~#
So, it's something in Apache/ZM but I have not nailed it down yet.