Camera goes blue screen

[wecoyote5]

Hello,

I am running Ubuntu 16.04 (64bit) and ZM1.30 with currently 2 Wansview Network cameras. Spent a lot of time reading though the documentation and got ZM installed correctly and setup to monitor the cameras. Both cameras and ZM work great; motion detect, record events, etc. but after a couple days one camera will go to a blue screen and stop detecting events. Logging into the camera from a mobile app shows the video just fine so I know the camera is still online. In the ZM logs I see that this particular camera goes from the max configured fps of 5 (which is set in the camera and not in ZM) to fps's in the 400-500s which obviously is a red flag to me. I haven't pinpointed when this change occurs yet as the log is huge, I'm still digging through that. I have to reboot the camera and restart ZM to get back to normal for a few more days. Both cameras are identical so the monitor setup is identical with the exception of the name and IP address. They are both setup for FFMpeg with no additional options (adding the "...types = video" breaks the connection altogether) and RTP/RTSP.

Has anyone come across this before and can point me where to debug further?
Thanks

[bbunge]

How much memory in your system and what resolution are your cameras set for?

[wecoyote5]

Hello,

Intel i5 quad core 2.5Ghz w/4GB and the resolution is 1920x1080, max fps is 5 configured in the camera. In ZM the load value is normally under 1.5, worst case I've seen it at 3 for a brief time. According to the 'ds -h' memory usage list /dev/shm is running at 42%. Based on what I've read I don't think the system is straining but maybe I'm missing something.

Thanks

[wecoyote5]

Hi all,

I have tried a few things but have been unsuccessful to stop the blue screen from happening. I've isolated the events in the log and was hoping someone may be able to shed some light.

Blue screen log.txt

(12.99 KiB) Downloaded 567 times

Any help is appreciated, thanks.

[bbunge]

Set the resolution in Zoneminder down by half and see if that helps. 960x540 based on what you have given us.

4 GIG of RAM may not be enough. Check what your database is using with mysqltuner. Look for memory used by innodb_buffer_pool With 4 GIG of RAM 2 GIG is being used for tmpfs which may be causing the system to use a lot of swap file.

[DervMan]

I've have a perfectly running ZM setup 1.28 for the last two years, however I managed to break the Debian OS so decided to re-install everything. Went with Ubuntu Server 16.10 and installed ZM 1.30 from the repo. Since doing that 2 of my three Tenvis cameras are showing a blue screen. The USB cams work a treat and I'm using the exact same camera config as before, having said that the ZM config options are a little different now. Nothing was restored it was all configured from scratch. Don't believe it's a shared memory issue as even with just one cam I still get the blue screen. Although all the cams are Tenvis they are different models so that's probably got some bearing on it.

I ran a tcpdump so 'see' what was happening at the network level and I can see the camera image being sent to ZM, but all ZM shows is a blue screen.

Sorry it's not much help but thought you might like to know it's not just you! See log below, I just get a continuous stream of Unable to Read content. Yet that same URL works from a browser on the zm box.

Code: Select all

root@HP-NAS:/var/log/zm# more zm_debug.log
11/01/16 15:06:00.829881 zms[13466].DB1-zm_logger.cpp/234 [LogOpts: level=DB1/DB1, screen=OFF, database=INF, logfile=DB1->/var/log/zm/zm_debug.log, syslog=INF]
11/01/16 15:06:00.829954 zms[13466].DB1-zm_utils.cpp/265 [Detected a x86\x86-64 processor with SSE3]
11/01/16 15:06:00.829977 zms[13466].DB1-zms.cpp/102 [Query: mode=jpeg&scale=100&maxfps=30&buffer=1000&monitor=7&connkey=745303&rand=1478012760]
11/01/16 15:06:00.830723 zms[13466].DB1-zm_monitor.cpp/2726 [Got 1 for v4l_captures_per_frame]
11/01/16 15:06:00.831623 zms[13466].DB1-zm_monitor.cpp/390 [monitor purpose=0]
11/01/16 15:06:00.831668 zms[13466].DB1-zm_monitor.cpp/398 [mem.size=61441760]
11/01/16 15:06:00.831677 zms[13466].DB1-zm_monitor.cpp/449 [Monitor Garage-IP has no zones, adding one.]
11/01/16 15:06:00.831690 zms[13466].DB1-zm_zone.cpp/51 [Initialised zone 0/All - 1 - 640x480 - Rgb:0000ff, CM:3, MnAT:15, MxAT:0, MnAP:50, MxAP:75000, FB:3x3, MnFP:50, MxFP:50000, Mn
BS:10, MxBS:0, MnB:0, MxB:0, OF: 0, AF: 0]
11/01/16 15:06:00.840128 zms[13466].DB1-zm_monitor.cpp/459 [Monitor Garage-IP has function 2]
11/01/16 15:06:00.840140 zms[13466].DB1-zm_monitor.cpp/460 [Monitor Garage-IP LBF = '%N - %Y-%m-%d %H:%M:%S %z', LBX = 0, LBY = 0, LBS = 1]
11/01/16 15:06:00.840147 zms[13466].DB1-zm_monitor.cpp/461 [Monitor Garage-IP IBC = 50, WUC = 25, pEC = 25, PEC = 25, EAF = 1, FRI = 1000, RBP = 6, ARBP = 6, FM = 0]
11/01/16 15:06:00.840156 zms[13466].DB1-zm_monitor.cpp/2989 [Loaded monitor 7(Garage-IP), 0 zones]
11/01/16 15:06:00.840192 zms[13466].DB1-zm_monitor.cpp/543 [Unable to map file /dev/shm/zm.mmap.7 (61441760 bytes) to locked memory, trying unlocked]
11/01/16 15:06:00.840207 zms[13466].DB1-zm_monitor.cpp/546 [Mapped file /dev/shm/zm.mmap.7 (61441760 bytes) to locked memory, unlocked]
11/01/16 15:06:00.840293 zms[13466].DB1-zm_stream.cpp/314 [We have obtained a lock on /var/run/zm/zms-745303.lock fd: 8]
11/01/16 15:06:01.786302 zms[13466].DB1-zm_monitor.cpp/3732 [Got QUERY command, sending STATUS]
11/01/16 15:06:02.884462 zms[13466].DB1-zm_monitor.cpp/3732 [Got QUERY command, sending STATUS]
11/01/16 15:06:03.676475 zms[13466].INF-zm_signal.cpp/40 [Got signal 15 (Terminated), exiting]
11/01/16 15:06:03.884859 zms[13466].DB1-zm_monitor.cpp/4201 [Cleaning swap files from /tmp/zm/zmswap-m7/zmswap-q745303]
11/01/16 15:06:03.884975 zms[13466].DB1-zm_logger.cpp/239 [Terminating Logger]



11/01/16 15:05:58.540461 zmc_m7[13452].ERR-zm_remote_camera_http.cpp/1141 [Unable to get response, disconnecting]
11/01/16 15:05:58.810927 zmc_m7[13452].ERR-zm_remote_camera_http.cpp/1040 [Unable to read content]
11/01/16 15:05:58.893898 zmc_m7[13452].ERR-zm_remote_camera_http.cpp/1141 [Unable to get response, disconnecting]
11/01/16 15:05:59.143935 zmc_m7[13452].ERR-zm_remote_camera_http.cpp/1040 [Unable to read content]
11/01/16 15:05:59.236137 zmc_m7[13452].ERR-zm_remote_camera_http.cpp/1141 [Unable to get response, disconnecting]
11/01/16 15:05:59.477401 zmc_m7[13452].ERR-zm_remote_camera_http.cpp/1040 [Unable to read content]
11/01/16 15:05:59.545472 zmc_m7[13452].ERR-zm_remote_camera_http.cpp/1141 [Unable to get response, disconnecting]

[mikb]

I had a similar problem with ZoneMinder and D-Link DCS900 cameras -- they would run fine for 2 months solid, then start getting the odd blue-screen interruption. Eventually the "odd interruption" became "unusable blueness"

I could see the cameras fine via Firefox, directly, but the stream to ZM wouldn't stay up.

ZM was frantically trying to re-get the camera, the camera was not playing ball.

In the end I concluded the camera was the problem, as restarting ZM didn't help, but restarting the camera DID.

So I now remote-reboot the cameras once a week to stop this happening. Never a problem since.

My hunch is that the camera firmware has a memory leak, and the first obvious casualty is the MJPEG stream/web handling, while the stills JPG/web access still works.

[bbunge]

I've have a perfectly running ZM setup 1.28 for the last two years, however I managed to break the Debian OS so decided to re-install everything. Went with Ubuntu Server 16.10 and installed ZM 1.30 from the repo. Since doing that 2 of my three Tenvis cameras are showing a blue screen. The USB cams work a treat and I'm using the exact same camera config as before, having said that the ZM config options are a little different now. Nothing was restored it was all configured from scratch. Don't believe it's a shared memory issue as even with just one cam I still get the blue screen. Although all the cams are Tenvis they are different models so that's probably got some bearing on it.

[/code]

Instructions for Ubuntu 16.10 in case you did not find them: https://wiki.zoneminder.com/Ubuntu_Serv ... e_easy_way

Running on my test machine with no issues...

[DervMan]

Thanks for the link bbunge, that's the guide I followed. Given all 4 cameras worked on 1.28 (1 still works on 1.30) and I can stream to the web browser on my PC I'm at a bit of a loss! I guess I'll need to buy more of the camera that works!

[wecoyote5]

Hi all,

Thanks bbunge I will check my SQL memory settings.
Devman, thanks for the validation, I'm already losing my mind and I don't need help . My system does do something slightly different, it can regain a connection but it report processing an excessive amount of fps (>50 when the cameras are set to 5) I've seen it sometimes in the hundreds.
Mikb, I've done the same thing in the meantime, I've set the cameras to automatically reboot 2 times a week.

[SteveGilvarry]

DervMan try ffmpeg, do you have your own topic I can help in? OP was using ffmpeg monitors yours are remote.

wecoyote5 put a MaxFPS into ZM just above the setting of the camera, stops it going completely nuts when it loses connection to the camera.
2016-10-30 21:44:10.228423 zmc_m3[17139].ERR-zm_ffmpeg_camera.cpp/280 [Unable to open input rtsp://admin: due to: Operation now in progress]
This tends to be something up with the camera, like I am already streaming and don't have capacity to accept another connection, but ZM doesn't seem to recover from it. All the pictures of Wansview shows them as wireless?
If you reboot the camera and change monitor function to None and back does that bring it back. If so then try just the function to None and back, check logs to see if you get that operation now in progress message. The try streaming camera from VLC while it is returning operation now in progress to see if that works.
Not really any solution from me but some avenues to hunt in.

[wecoyote5]

Hi all,

Thanks Steve, I tried that setting in ZM first thinking maybe my camera was not truly limiting and it really threw things into a snit, the frame rate went through the roof (>300fps) and the connection got dropped, etc; it threw all kinds of errors. This needed a camera reboot and a ZM restart.
Yes the cameras are wireless but I have a dedicated wi-fi access point and RF channel just for the video streams so I know I do not have other devices on the same wireless nor do I have interference. Also, during this 'outtage' I can successfully connect to the camera with my phone and/or PC and the video is fine.
I will try toggling the camera function and see what happens; good thought about using VLC, I will try that too. I used VLC initially to make sure that I got the syntax to the RSTP stream correct before trying ZM. THis will at least point the debug effort to the camera vs. ZM.

bbunge, I ran mysqltuner and this is the innodb info:
-------- InnoDB Metrics -----------------------------------------------------
[--] InnoDB is enabled.
[OK] InnoDB buffer pool / data size: 128.0M/24.8M
[OK] InnoDB buffer pool instances: 1
[!!] InnoDB Used buffer: 52.40% (4292 used/ 8191 total)
[OK] InnoDB Read buffer efficiency: 100.00% (212031583 hits/ 212034398 total)
[!!] InnoDB Write buffer efficiency: 0.00% (0 hits/ 1 total)
[OK] InnoDB log waits: 0.00% (0 waits / 269827 writes)

Thanks again all!