Lost all network cameras

[rip]

So, I can't get network cameras to stream or capture after going from 1.28.1 to 1.29.
UBUNTU 14.04 server using PPA iconnor:zoneminder (not master).

I've spent DAYS on this, and tonight what I did was, stop my processes, rename my original database [so I have a reference for every single setting zm renamed to zm_original and zm dropped after purge], removed the PPA, deleted all my events, etc, purged zoneminder and did a pretty near clean install. Even removed the PPA and re-added it just to follow the steps close as possible.

I did not change the ScriptAlias, but did verify that that path really exists and nph-zms is where I expect it to be.

apache2.conf is nearly bone stock (simple entry for phpmyadmin added after this started so I could easily view the db for sanity checking)

I was careful with all the steps in the 1.29 the easy way doc, explicitly following it and pretending I know nothing so I didn't second guess it on the reinstall. a2enmods and a2enconf, etc., all done.

I did have an early issue with socks files (/run/zm) hung up and stopped apache2 and zoneminder and nuked those manually and restarted. Quieter logs but same result. The backend is processing [capturing at a few FPS and blue check signal background]. 858 card (PV-183) based cams are OK and stream and capture.

I suspect something, somewhere in Apache is at blame, since purging zm and rebuilding gave the same result. So please, if someone can point out the obvious and make me look like an idiot, that'd be great. I'm generally pretty good at this kind of thing, but kind of burned out on it after a few nights hacking on it. Been meaning to upgrade off 14.04 but rather wait for 16.04 LTS so don't want to nuke it just yet.

This is a headless server for what it's worth. No desktop or monitor to browse localhost.

PATH_ZMS is /zm/cgi-bin/nph-zms

/zm/api is all green

Code: Select all

getVersion.json
{
    "version": "1.29.0",
    "apiversion": "1.0"
}

Code: Select all

Clean start apache2 error.log:
[Fri Feb 19 23:54:53.114751 2016] [mpm_prefork:notice] [pid 1736] AH00163: Apache/2.4.7 (Ubuntu) PHP/5.5.9-1ubuntu4.14 configured -- resuming normal operations
[Fri Feb 19 23:54:53.114806 2016] [core:notice] [pid 1736] AH00094: Command line: '/usr/sbin/apache2'

Code: Select all

zmc_m3 [Network Camera, can view in browser and worked in 1.28.1, etc.]
016-02-20 00:01:51.527192	zmc_m3		1642	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-02-20 00:01:51.472337	zmc_m3		1642	ERR	Unable to read content	zm_remote_camera_http.cpp	1040
2016-02-20 00:01:51.082095	zmc_m3		1642	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-02-20 00:01:51.033967	zmc_m3		1642	ERR	Unable to read content	zm_remote_camera_http.cpp	1040
2016-02-20 00:01:50.647195	zmc_m3		1642	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-02-20 00:01:50.595472	zmc_m3		1642	ERR	Unable to read content	zm_remote_camera_http.cpp	1040
2016-02-20 00:01:50.303020	zmc_m3		1642	ERR	Unable to get response, disconnecting	zm_remote_camera_http.cpp	1141
2016-02-20 00:01:50.157137	zmc_m3		1642	ERR	Unable to read content	zm_remote_camera_http.cpp	1040

zma_m3 is quiet since the last reboot. Again, m3 is the first network camera.

Code: Select all

syslog mirrors zmc_m3:
Feb 20 00:46:35 server1 zmc[1642]: ERR [Unable to get response, disconnecting]
Feb 20 00:46:35 server1 zmc[1642]: ERR [Unable to read content]
Feb 20 00:46:35 server1 zmc[1642]: ERR [Unable to get response, disconnecting]
Feb 20 00:46:36 server1 zmc[1642]: ERR [Unable to read content]
Feb 20 00:46:36 server1 zmc[1642]: ERR [Unable to get response, disconnecting]
Feb 20 00:46:36 server1 zmc[1642]: ERR [Unable to read content]
Feb 20 00:46:36 server1 zmc[1642]: ERR [Unable to get response, disconnecting]
Feb 20 00:46:37 server1 zmc[1642]: ERR [Unable to read content]
Feb 20 00:46:37 server1 zmc[1642]: ERR [Unable to get response, disconnecting]

dmesg is quiet
mysqld/error.log is quiet

thanks!

[bbunge]

You said that you upgraded but inferred that you followed the "Easy Way" instructions? Those are for install.

The upgrade instructions are located here:
https://wiki.zoneminder.com/Common_Issu ... PA_version

May make a difference..

There is also something at the end you need to do which some folks overlook:

Open Zoneminder. Click on Options - Paths

Change PATH_ZMS to /zm/cgi-bin/nph-zms

[rip]

Thanks for the reply.

I originally did upgrade and followed the upgrade instructions. Then spent 2 days trying to get IP cams to work. I read tons of stuff all over the net, and made sure the db looked normal, etc. I really did try hard and usually good at hammering these things out.

So I couldn't get it ironed out. OK, no big deal, lets stop the processes, purge zoneminder, drop the zm database, removed the PPA [thought I might of added master on accident, nope, it was good, but still removed it], rm -Rf * all my events and images, etc., made sure .conf files were gone (zoneminder.conf for apache2, zm.conf.) I did allow the timezone in php.ini to remain across tries.

Then ran through the steps of a clean install, which was find with a couple expected bumps (like, a2enmod cgi is already enabled, etc. No harm running the command again.)

Path-zms was one of the first things I listed in my post on purpose.

[bbunge]

Hmm... I ran the install procedure (from the WIKI) on a fresh Ubuntu 14.04 server install this afternoon and everything worked as it was supposed to work. I did the install from another PC and just copied and pasted the commands into the Putty terminal. You were getting a response from the API so that works. Usually if you do not get video the CGI was not enabled or the PATH_ZMS is wrong.

Could be some permissions: Set permissions of /etc/zm/zm.conf to root:www-data 740

chmod 740 /etc/zm/zm.conf

chown root:www-data /etc/zm/zm.conf

[rip]

It's not that either. Still a complete mystery to me. I haven't had time to mess it with much. probably will dump the whole things and try again.

[iconnor]

Those error logs show a failure to grab images... you will need to crnak up the debug logging options to at least 3 and see what it logs.

[rip]

Those error logs show a failure to grab images... you will need to crnak up the debug logging options to at least 3 and see what it logs.

Thanks for the reply Isaac. This has be stumped. 12.04 with 1.28.1 worked.

I put the debug at 4 in the snippets below.

I did a complete reload, multiple times, now using 15.10 (from fresh ISO download) and the 15.04/15.10 "the easy way" instructions.

You'll notice that HTTP 1.0 vs 1.1 in the log below, but I've tried both ways. I also thought maybe SHM was running out of memory but it's OK.

I've made sure apparmor was off, just in case.

ZM_PATH is correct at: /zm/cgi-bin/nph-zms

But it's apparently not making Sockets right. Local analog cameras on a 858 card are fine but IP cam is not.

Code: Select all

rip@server1:/run/zm$ ls -la
total 4
drwxr-xr-x  2 www-data www-data  80 Feb 27 22:11 .
drwxr-xr-x 21 root     root     660 Feb 27 22:10 ..
srwxr-xr-x  1 www-data www-data   0 Feb 27 22:11 zmdc.sock
-rw-r--r--  1 www-data www-data   4 Feb 27 22:11 zm.pid

Apache error log:

Code: Select all

Sat Feb 27 22:04:12.924089 2016] [:error] [pid 1055] [client 10.168.1.46:54554] ERR [socket_sendto( /var/run/zm/zms-407307s.sock ) failed: No such file or directory], referer: http://10.168.1.151/zm/?view=watch&mid=1

It looks like it's (trying to anyway) pumping data:

Code: Select all

02/27/16 23:16:27.186517 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/124 [Connected to host, socket = 6]
02/27/16 23:16:27.186548 zmc_m1[1607].DB2-zm_remote_camera_http.cpp/138 [Sending request: GET /image?res=full&x0=0&y0=0&x1=2048&y1=1536&quality=6&doublescan=0 HTTP/1.1
User-Agent: ZoneMinder/1.29.0
Host: 10.168.1.163

]
02/27/16 23:16:27.186577 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/147 [Request sent]
02/27/16 23:16:27.189408 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/207 [Just getting 15]
02/27/16 23:16:27.189438 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/210 [Expecting 15 bytes]
02/27/16 23:16:27.189457 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/232 [Read 15 bytes]
02/27/16 23:16:27.189468 zmc_m1[1607].DB4-zm_remote_camera_http.cpp/238 [HTTP/1.0 200 OK]
02/27/16 23:16:27.189478 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/862 [Unable to extract entire header from stream, continuing]
02/27/16 23:16:27.193178 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/207 [Just getting 623]
02/27/16 23:16:27.193208 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/210 [Expecting 623 bytes]
02/27/16 23:16:27.193227 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/232 [Read 623 bytes]
02/27/16 23:16:27.193238 zmc_m1[1607].DB4-zm_remote_camera_http.cpp/238 [HTTP/1.0 200 OK
Content-Type: image/jpeg

▒▒▒▒]
02/27/16 23:16:27.193255 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/773 [Got status '200' (OK), http version 1.0]
02/27/16 23:16:27.193267 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/814 [Got content type 'image/jpeg']
02/27/16 23:16:27.193608 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/207 [Just getting 1460]
02/27/16 23:16:27.193622 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/210 [Expecting 1460 bytes]
02/27/16 23:16:27.193635 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/232 [Read 1460 bytes]
02/27/16 23:16:27.193645 zmc_m1[1607].DB4-zm_remote_camera_http.cpp/238 [▒▒▒▒]
02/27/16 23:16:27.194097 zmc_m1[1607].DB3-zm_remote_camera_http.cpp/207 [Just getting 1460]

I've run zm for a few years on usually ubuntu distros, so like to think I'm a little literate about it. Been using linux since beta 16 when Linus was still writing it. I'm not an apache expert though, I make equipment for satellite uplinks.

[bbunge]

What are the cameras you are using (I looked through the thread and could not see that you mentioned them)? The errors you are seeing tell me that you have a camera setting not just right. Ubuntu 15.10 should run like a champ! You may want to do teh setup with the default ZM 1.28.1 get your cameras working then upgrade.

bb

[rip]

The one I'm testing against is a 3mp Arecont 3155DN. I actually exported my camera settings table before all this so I didn't loose my strings, so I know exactly how they were set up. (arecont options can make for hairy path strings.)

Using that info, I can pull images and streams from VLC on my windows machine. I can ping the camera from the ZM machine (Its headless 6 core with 16gb ram so plenty of horsepower.)

I spent a lot of yesterday trying different camera settings, thinking maybe something in zm changed breaking how it was.

The way I was doing it was remote and grabbing individual full frame images (2048x1536) with ZM, as this game me more control over the image than streaming. I even went to the camera itself and took apart the user interface to look at the static and streaming URLs (wrapped in a iframe in the UI) to compare and use try in ZM.

It's probably something stupid and I've been looking at it so long I can't see it. :/ Hate it when that happens.

[bbunge]

Did you upgrade your database and permissions?

Change permissions in your "ZM database to allow the creation of a table

mysql -uroot -p -e "grant all on zm.* to 'zmuser'@localhost identified by 'zmpass';"

Run the database upgrade

/usr/bin/zmupdate.pl

[rip]

Ummm, yeah....I formatted the entire system twice and installed from the ground up on 15.10 since multiple tries on 14.04 were not working.

I did do the DB upgrade on the previous tries before wiping it out, and installing (new partition and everything....even tried a couple ways because I was grasping at anything, like possibly LVM was somehow tripping up permissions. Nope.)

Everything works except IP cams.

[mcouture]

I had 1.29.0 running for awhile now on Unbuntu and I run the Software Updater and it showed a bunch of updates available, including ZoneMinder.


I ran the updater and now all my cameras do not connect (all wifi/network cams)

[mcouture]

I had to run the "zmupdate.pl -f" command.

[bbunge]

From the WIKI...

Guides for Ubuntu Desktop

Installing Zoneminder on a desktop system can have its benefits. It can also pose problems!

If you use one of the desktops provided with Ubuntu (Gnome, Unity, KDE, et. al.) use the corresponding Ubuntu server install/upgrade instructions. You may need to install tasksel to make it easier to install LAMP as most of the Ubuntu instructions start with a system set up with LAMP. It is recommended to install Zoneminder from a terminal. Using a software/package manager can get you in trouble!

If you are using a distro such as Mint, find out which Ubuntu system your version is based on and use the corresponding Ubuntu server install/upgrade. And do the procedure from a terminal!

Oh, did I fail to mention it is easy to copy the command instructions from the instructions on the WIKI and paste them into the terminal window? It is a great way to avoid typos!!!

Repeat: Using a software/package manager can get you in trouble!

[rip]

This is 15.10 x64 server, headless, no gui. Multiple fresh installs, configured via ssh as root.

Something in the sockets broke, and it's none of the usual suspects.

PATH_ZMS is /zm/cgi-bin/nph-zms before we go over that again. Going to IP/zm/cgi-bin/nph-zms does not give a 404, just blank

Some of the below is actually trying to grab /img.jpg from the cam, which is a static jpg no stream to error. It works in a browser on my network (windows and linux machines, but the WEB server can't pull it.

Code: Select all

root@server1:/var/log# cat  /var/log/syslog |grep .sock
Mar  1 12:51:16 server1 web_php[9546]: ERR [socket_sendto( /var/run/zm/zms-997856s.sock ) failed: No such file or directory]
Mar  1 15:58:27 server1 web_php[10351]: ERR [socket_sendto( /var/run/zm/zms-219461s.sock ) failed: No such file or directory]
Mar  1 15:58:28 server1 web_php[10351]: ERR [getCmdResponse stream error: socket_sendto( /var/run/zm/zms-219461s.sock ) failed: No such file or directory - checkStreamForErrors()]
Mar  1 20:36:44 server1 web_php[10171]: ERR [socket_sendto( /var/run/zm/zms-253299s.sock ) failed: No such file or directory]
Mar  1 20:36:44 server1 web_php[10171]: ERR [getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-253299s.sock ) failed: No such file or directory - checkStreamForErrors()]
Mar  1 20:37:27 server1 web_php[10351]: ERR [socket_sendto( /var/run/zm/zms-787429s.sock ) failed: No such file or directory]
Mar  1 20:37:27 server1 web_php[10351]: ERR [getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-787429s.sock ) failed: No such file or directory - checkStreamForErrors()]
Mar  1 20:41:34 server1 web_php[10564]: ERR [socket_sendto( /var/run/zm/zms-815036s.sock ) failed: No such file or directory]
Mar  1 20:41:34 server1 web_php[10564]: ERR [getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-815036s.sock ) failed: No such file or directory - checkStreamForErrors()]
Mar  1 20:43:29 server1 web_php[10169]: ERR [socket_sendto( /var/run/zm/zms-684125s.sock ) failed: No such file or directory]
Mar  1 20:43:29 server1 web_php[10169]: ERR [getStreamCmdResponse stream error: socket_sendto( /var/run/zm/zms-684125s.sock ) failed: No such file or directory - checkStreamForErrors()]
root@server1:/var/log# 

Code: Select all

root@server1:/var/run/zm# ls -la
total 4
drwxr-xr-x  2 www-data www-data  80 Mar  1 20:43 .
drwxr-xr-x 22 root     root     740 Mar  1 21:17 ..
srwxr-xr-x  1 www-data www-data   0 Feb 28 15:44 zmdc.sock
-rw-r--r--  1 www-data www-data   4 Feb 28 15:44 zm.pid
root@server1:/var/run/zm# 

But I can WGET the image on the server:

Code: Select all

root@server1:~# wget http://10.168.1.163/img.jpg
--2016-03-01 21:34:39--  http://10.168.1.163/img.jpg
Connecting to 10.168.1.163:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [image/jpeg]
Saving to: ‘img.jpg’

img.jpg                 [ <=>                  ] 210.17K  --.-KB/s   in 0.05s  

2016-03-01 21:34:39 (4.32 MB/s) - ‘img.jpg’ saved [215213]

root@server1:~# ls -la *.jpg
-rw-r--r-- 1 root root 215213 Mar  1 21:34 img.jpg
root@server1:~# 

So, it's something in Apache/ZM but I have not nailed it down yet.