IP sources keep going red...

elfking · Post by **elfking** » Mon Jan 11, 2010 5:26 pm

I have been trying to get 1.24.2 working correctly for me and have been running into the issue that randomly about two camera's will drop (turn red) durring the course of a few days. The camera's are working fine, I have an older version of zoneminder on a different machine running 1.22.3 which can reach the same cameras just fine in the same way without having any sources turning red.

I am currently running:
OS: Debian 5.03 (Lenny)
Kernel Linux 2.6.26-2-686
GNOME 2.22.3
Motherboard: Tyan S2895
CPU0: AMD Opteron 250
CPU1: AMD Opteron 250
Ram: 2GB

I have tried this same setup on an older Intel P4 setup and having identical results with this red source issue. I have also tried this with the Arch Linux image with the same issue...

The cameras are all Axis IP cameras, a few different models, 206W, 207, 221, 212.

Again please keep in mind all of them are working fine on my older 1.22.3 zoneminder box...

Log files:

/var/log/messages |grep m9
Jan 11 03:06:44 debian-zm2 zmc_m9[27300]: INF [New Debug Level = 5, New Debug Log = /tmp/zm_debug.log.27300]
Jan 11 03:06:44 debian-zm2 zmc_m9[27300]: INF [Starting Capture]
Jan 11 03:10:18 debian-zm2 zma_m9[24404]: INF [1st_Floor_-_WEST_ENTRANCE_INSIDE: 513000 - Processing at 2.19 fps]
Jan 11 03:13:46 debian-zm2 zmc_m9[27300]: INF [1st_Floor_-_WEST_ENTRANCE_INSIDE: 1000 - Capturing at 2.37 fps]
Jan 11 03:17:21 debian-zm2 zma_m9[24404]: INF [1st_Floor_-_WEST_ENTRANCE_INSIDE: 514000 - Processing at 2.36 fps]
Jan 11 03:21:00 debian-zm2 zmc_m9[27300]: INF [1st_Floor_-_WEST_ENTRANCE_INSIDE: 2000 - Capturing at 2.30 fps]
Jan 11 03:24:35 debian-zm2 zma_m9[24404]: INF [1st_Floor_-_WEST_ENTRANCE_INSIDE: 515000 - Processing at 2.30 fps]
Jan 11 03:28:16 debian-zm2 zmc_m9[27300]: INF [1st_Floor_-_WEST_ENTRANCE_INSIDE: 3000 - Capturing at 2.29 fps]
Jan 11 03:32:02 debian-zm2 zma_m9[24404]: INF [1st_Floor_-_WEST_ENTRANCE_INSIDE: 516000 - Processing at 2.24 fps]
Jan 11 03:35:43 debian-zm2 zmc_m9[27300]: INF [1st_Floor_-_WEST_ENTRANCE_INSIDE: 4000 - Capturing at 2.24 fps]
Jan 11 03:36:03 debian-zm2 zmc_m9[27300]: INF [Got signal 15 (Terminated), exiting]
debian-zm2:/tmp#

when I try and look for the PID for my two cameras that are red, I don't see any related monitors running.. so it seems they crash, never to restart on their own.. (requires a zm restart or manual disable/enable camera)

debian-zm2:/tmp# ps axf | grep zmc
24361 ? S 388:04 \_ /usr/local/bin/zmc -m 1
24365 ? S 415:23 \_ /usr/local/bin/zmc -m 2
24377 ? S 396:21 \_ /usr/local/bin/zmc -m 5
24383 ? S 382:52 \_ /usr/local/bin/zmc -m 6
24389 ? S 406:13 \_ /usr/local/bin/zmc -m 7
24395 ? S 400:45 \_ /usr/local/bin/zmc -m 8
24413 ? S 410:22 \_ /usr/local/bin/zmc -m 11
24855 ? S 352:55 \_ /usr/local/bin/zmc -m 16
27058 ? S 83:26 \_ /usr/local/bin/zmc -m 4
27741 ? R 0:07 \_ /usr/local/bin/zmc -m 3
27751 pts/1 S+ 0:00 \_ grep zmc
debian-zm2:/tmp#

The red monitors are 9 and 10 both show this in the zdmc.log
-----------------------------------------------------------------------------------------
debian-zm2:/tmp# more zmdc.log |grep 'zmc -m 9'
01/11/10 00:19:29.586704 zmdc[24339].INF [Starting pending process, zmc -m 9]
01/11/10 00:19:29.587624 zmdc[24339].INF ['zmc -m 9' starting at 10/01/11 00:19:29, pid = 27237]
01/11/10 01:19:43.612925 zmdc[24339].ERR ['zmc -m 9' exited abnormally, exit status 255]
01/11/10 01:19:43.613294 zmdc[24339].INF [Starting pending process, zmc -m 9]
01/11/10 01:19:43.641876 zmdc[24339].INF ['zmc -m 9' starting at 10/01/11 01:19:43, pid = 27264]
01/11/10 01:48:14.394717 zmdc[24339].ERR ['zmc -m 9' exited abnormally, exit status 255]
01/11/10 01:48:14.395011 zmdc[24339].INF [Starting pending process, zmc -m 9]
01/11/10 01:48:14.403105 zmdc[24339].INF ['zmc -m 9' starting at 10/01/11 01:48:14, pid = 27274]
01/11/10 03:06:44.185115 zmdc[24339].INF ['zmc -m 9' stopping at 10/01/11 03:06:44]
01/11/10 03:06:44.206200 zmdc[24339].ERR ['zmc -m 9' exited abnormally, exit status 255]
01/11/10 03:06:44.306610 zmdc[24339].INF [Starting pending process, zmc -m 9]
01/11/10 03:06:44.315094 zmdc[24339].INF ['zmc -m 9' starting at 10/01/11 03:06:44, pid = 27300]
01/11/10 03:36:03.263298 zmdc[24339].INF ['zmc -m 9' stopping at 10/01/11 03:36:03]
01/11/10 03:36:03.287721 zmdc[24339].ERR ['zmc -m 9' exited abnormally, exit status 255]

-----------------------------------------------------------------------------------------

debian-zm2:/tmp# tail -n 100 zm_debug.log.27300
01/11/10 03:36:03.047362 zmc_m9[27300].DB3-zm_remote_camera_http.cpp/171 [Expecting 1448 bytes]
01/11/10 03:36:03.047407 zmc_m9[27300].DB3-zm_remote_camera_http.cpp/196 [Read 1448 bytes]
01/11/10 03:36:03.049194 zmc_m9[27300].DB3-zm_remote_camera_http.cpp/171 [Expecting 807 bytes]
01/11/10 03:36:03.049219 zmc_m9[27300].DB3-zm_remote_camera_http.cpp/196 [Read 807 bytes]
01/11/10 03:36:03.049232 zmc_m9[27300].DB3-zm_remote_camera_http.cpp/937 [Got end of image by length, content-length = 48421]
01/11/10 03:36:03.049268 zmc_m9[27300].DB3-zm_remote_camera_http.cpp/1004 [Returning 48421 (48421) bytes of captured content]
01/11/10 03:36:03.070789 zmc_m9[27300].DB3-zm_remote_camera_http.cpp/105 [Connected to host, socket = 4]
01/11/10 03:36:03.070850 zmc_m9[27300].DB3-zm_remote_camera_http.cpp/126 [Request sent]
01/11/10 03:36:03.263563 zmc_m9[27300].INF-zm_signal.cpp/43 [Got signal 15 (Terminated), exiting]
01/11/10 03:36:03.263617 zmc_m9[27300].ERR-zm_remote_camera_http.cpp/146 [Select error: Interrupted system call]
01/11/10 03:36:03.263641 zmc_m9[27300].ERR-zm_remote_camera_http.cpp/577 [Unable to read header]
01/11/10 03:36:03.263659 zmc_m9[27300].ERR-zm_remote_camera_http.cpp/1043 [Unable to get response]
01/11/10 03:36:03.263709 zmc_m9[27300].ERR-zmc.cpp/260 [Failed to capture image from monitor 9 (0/1)]
debian-zm2:/tmp#

-----------------------------------------------------------------------------------------

Anyone have any thoughts? I have restarted the zm service to get these two cameras going again and will also check and make sure its not the SAME cameras showing red.. on my past two machines they were different...

Thank you for your time,
-Brian

kingofkya · Post by **kingofkya** » Tue Jan 12, 2010 4:53 am

try running ping to all the cams for a few hours. you will probbly see it misses packets at the same time. For me the problem lay in a crappy switch. So I would try that first.

nicocam · Post by **nicocam** » Tue Jan 12, 2010 2:11 pm

We used to have the same problem, and the reason was a crappy link to the cameras.

If delays to the camera are big the monitor becomes really unstable. When we fixed the delays the problem disappeared.

The version we are using is 1.24.2, we've never tested another. We are using trendnet tv-ip110-ish cameras, (problems are worse using wireless cameras).

What surpises me is that you say you don't have the problem using 1.22.3, why is that?, perhaps something in the way that it "times out" the cameras changed, don't know.

elfking · Post by **elfking** » Thu Jan 14, 2010 10:17 pm

Thank you for your replies, I have a couple screenshots which shows at the same time the newer version shows the source red, older one shows it fine... 0 packet loss (quick ping check) I am running a long one that has been going since this morning and will keep an eye on it..

So the info I found in the log files just pretty much says that it can't reach the camera? Does it ever try again after it times out? Say the camera doesn't respond for a length of time and comes back.. shouldn't the service check every once in a while and go back to green when it returns?

Thanks again for the help,
-Brian

Camera labeled "1st_Floor_WEST_ENTRANCE" is the one we're looking at...

Current version (problem setup...)

(Older working fine, on the same network.. same cameras)

kingofkya · Post by **kingofkya** » Sat Jan 16, 2010 5:44 am

try
ping -t 192.168.XXX.XXX

then it will keep going so you can see what is going on when it drops

jfkastner · Post by **jfkastner** » Sat Jan 16, 2010 2:53 pm

try a lower fps setting

i have 11 ip-cams but a few are in darker areas and provide only about half the fps than the others - makes sense to me, small lense/chip needs a long exposure to make up for low light

seems to me ZM does not like it at all when it tries to poll cams and doesn't get the pic ... ZMC stalls/restarts and the cam 'turns red'

if your axis is a wifi try a different wifi channel, your ping times are all different ... or bad wiring/interferrence in the wires

kp4djt · Post by **kp4djt** » Sun Jan 17, 2010 5:13 am

Mine was never that bad, but I did see artifacts that looked like slow response.

Here is what I did, I added a 2nd Gigabit NIC card to the ZM box, the cams all talk to the ZM box on that NIC the primary NIC is used for network access. I put all of the cams on a gigabit switch, though the cams are only 100MB links the link between the switch and the host is Gigabit. That made
a big difference. My data rate on the camera port jumped from 300KB/s to well over 1MB/s. I saw no more artifacts from delayed frames.

Oh yes I also added DNSMASQ and let it assign the IP addresses to the cams, I listed the IP addresses for each cam in the config file that way I do not have to mess around with the cams, when I add a new cam to the network I put it's MAC address in the dnsmasq file and away it goes.

Moving to Gigabit Ethernet was a big help. I think if I go to a more expensive switch I will see even more gains. I am using a 8 port Netgear switch at this time.

troyy0206 · Post by **troyy0206** » Tue Feb 02, 2010 10:16 pm

kp4djt wrote:Mine was never that bad, but I did see artifacts that looked like slow response.

Here is what I did, I added a 2nd Gigabit NIC card to the ZM box, the cams all talk to the ZM box on that NIC the primary NIC is used for network access. I put all of the cams on a gigabit switch, though the cams are only 100MB links the link between the switch and the host is Gigabit. That made
a big difference. My data rate on the camera port jumped from 300KB/s to well over 1MB/s. I saw no more artifacts from delayed frames.

Oh yes I also added DNSMASQ and let it assign the IP addresses to the cams, I listed the IP addresses for each cam in the config file that way I do not have to mess around with the cams, when I add a new cam to the network I put it's MAC address in the dnsmasq file and away it goes.

Moving to Gigabit Ethernet was a big help. I think if I go to a more expensive switch I will see even more gains. I am using a 8 port Netgear switch at this time.

Just curious, what are the basic hardware specs on your server and how many cameras? Also, what brand of cams?

Thanks,
Troy

kp4djt · Post by **kp4djt** » Tue Feb 02, 2010 10:27 pm

the MoBo is a G31M3 V2. it has a dual core, I think it is 2Ghz, or there abouts, I will have to dig in and see what it is. It has 2G of memory and a 500G Hd. The camera nic card is a gigabit card, and it talks to a Netgear 8 port gigaswitch. The cameras are D-link 910 and a 920, there is also a Trendnet TV-IP110. They all work quite well for what they cost (I think it was between USD 68 and 72. The 920 was I believe USD 82.

You can go have a look at it at the following address
www.wchast.com/zm
login = guest
pass = guest

kp4djt · Post by **kp4djt** » Tue Feb 02, 2010 10:29 pm

There are 6 cams. The Wifi cam (D-Link 920) is used for video conf, or a rapid deploy to some location on the property where I am trying to capture something I can never be there to catch.

jimmyzshack · Post by **jimmyzshack** » Sun Mar 14, 2010 1:28 pm

I have this same problem. I have bought a new wireless router and still it happens but it's proable bc of these cheap cameras.

My questioin is does any know what the html code is that is sent when you press save in the source window? that always brings the camera's back up so i want to use curl and make a cron job to do this.

jfkastner · Post by **jfkastner** » Sun Mar 14, 2010 2:31 pm

install wireshark or any other packet capturing program and look what's going on

especially with wifi problems you are able to see what/how many packets are truncated or have a wrong crc etc

kp4djt · Post by **kp4djt** » Sun Mar 14, 2010 5:08 pm

My system has been going along without giving me this problem ever since I found a bad switch on my network. Thing was dropping frames, once fixed things have gone along quite well, until yesterday.

I have a camera I am using to get a idea of what I am going to have to do to capture auto plates out at the entry way to my property. I am located such that I have a choke point for vehicles entering or leaving the neighborhood. We had a break in at a neighbors house a few weeks ago, we were able to make the vehicle but not that plates.

I have had this camera set up now for about a week. It has done very well up until yesterday. The camera is a D-Link 920 wifi cam. it was running a flat 5fps, which appeared to be a good speed to capture images that would with a better camera give readable plates.

Night before last we had a cold front go through here, yesterday broke very windy, the trees all in motion. I noticed that the "function" and "source" had turned red, I went in and took it off line and then back on line. I sat there for a bit and then went back off line. I then went to monitor the live video clicking on the camera name. I observed that the speed was dropping fro 5fps to about 1 or 1.5 fps and finally it would freeze. Went to the log and observed the following:

Mar 13 17:24:11 zmhost01 httpd: socket_sendto( /var/lib/zm/socks/zms-989222s.sock ) failed: No such file or directory
Mar 13 17:24:11 zmhost01 httpd: array (
Mar 13 17:24:11 zmhost01 0: =>
Mar 13 17:24:11 zmhost01 array: (
Mar 13 17:24:11 zmhost01 'file': => '/var/lib/zm/www/ajax/stream.php',
Mar 13 17:24:11 zmhost01 'line': => 55,
Mar 13 17:24:11 zmhost01 'function': => 'ajaxError',
Mar 13 17:24:11 zmhost01 'args': =>
Mar 13 17:24:11 zmhost01 array: (
Mar 13 17:24:11 zmhost01 0: => 'socket_sendto( /var/lib/zm/socks/zms-989222s.sock ) failed: No such file or directory',
Mar 13 17:24:11 zmhost01 ),:
Mar 13 17:24:11 zmhost01 ),:
Mar 13 17:24:11 zmhost01 1: =>
Mar 13 17:24:11 zmhost01 array: (
Mar 13 17:24:11 zmhost01 'file': => '/var/lib/zm/www/index.php',
Mar 13 17:24:11 zmhost01 'line': => 116,
Mar 13 17:24:11 zmhost01 'args': =>
Mar 13 17:24:11 zmhost01 array: (
Mar 13 17:24:11 zmhost01 0: => '/var/lib/zm/www/ajax/stream.php',
Mar 13 17:24:11 zmhost01 ),:
Mar 13 17:24:11 zmhost01 'function': => 'require_once',
Mar 13 17:24:11 zmhost01 ),:
Mar 13 17:24:11 zmhost01 ):

From previous experience when I see one of these either I have a network problem or a memory problem. This one I really suspected was a network problem, I started a ping session and observed that when either the camera started to slow down my ping RTT had gone up quite a bit, and if it went up to quite long periods I would see the above in the log file. So I knew pretty much that my WiFi was choking things. I went in and dropped my frame rate down to 3fps, and loosened up the zones a bit, that appeared to have appeased the network god so things are back to normal, even tough the wind is back up today I still have not had any big issues with it, I do see one of these every once in a while:

Mar 14 12:53:44 zmhost01 zmwatch[2999]: INF [Restarting capture daemon for Tag_Capture, time since last capture 8 seconds (1268585624-1268585616)]

But it appears to be recovering. Part of the problem is distance, the cam is about 300 ft from the access point, and I think that a large load of frames of video get trashed and ZM gets konfused. Slowing things down seems to have fixed it.

jfkastner · Post by **jfkastner** » Mon Mar 15, 2010 12:16 am

wind can be indeed a factor in wireless

swaying trees, leafes, signs, powerlines, debris etc can all reflect/dampen the 2.4GHz waves from your wifi and since the timing is critical even a small distortion might disrupt it

you might try to play with the antennas - i had to boost mine a bit - and removed the outer plastic layer from the actual metal that's inside, so that the waves don't have to pass thru anything while leaving the access point

since it's illegal in some countries to boost transmit power you can just make it work more efficiently

do NOT change the length though that'll modify the optimum freq for this antenna (shorter => higher freq)

kp4djt · Post by **kp4djt** » Mon Mar 15, 2010 12:33 am

Actually I am looking at building some patch antennas for my wifi stuff, I will put the patch antennas on the camera end aimed at the AP. I just have to get me some pcb with the proper sub-strait, Then I will build me the patch antennas and put one with each camera, aimed at the AP, that should take care of it. In this case there are several things against it,
1. Distance
2. vegetation,
3. cross polarised antenna.

#3 is because I could not get the antenna vertical in the case the camera is in, the patch will take care of that. And it should give me enough fade margin to overcome the other two issues.