Problem with corrupt JPEG images

[gregom]

I have a new build i'm testing, with new cameras as well. I am running ZM 1.27 on Ubuntu 12.04.3 LTS. The server has two quad-core Xeon E5606 processors and 16 GB DDR3 ECC memory, 18 TB RAID-5 storage for events. Load likes to hang out around 2 to 3. Highest i've seen is 3.43. It isn't dropping frames or events or anything like that. Log is green.

On this test build I'm recording 8 Vivotek IP8362 cams at 1080p 5 fps, and 10 Axis M3533 cams at 800x600 5 fps. All is well except one of my Vivotek's is starting to get corrupt JPEG images recorded. These look very similar to the corrupt JPEGs I have been getting from my old Toshiba IK-WB70A cameras (still in use on other DVR servers). I suspect the issue is the camera and not ZM, but I wanted to see if anyone else is experiencing similar issues. I saw a thread from about five months ago that shows a similar JPEG corruption issue: http://www.zoneminder.com/forums/viewto ... rupt+video Maybe they are related... I don't know but i'm trying to find the cause of this.

I also have 34 other IP cameras in use on my other (five) DVR servers (older hardware, ZM 1.23.3 on Ubuntu 8.04 LTS for the last five years) and the issue exists on those as well but only for my Toshiba IK-WB70A cameras. No other cameras do this... I have 20 Toshiba IK-WD01A's that don't have this issue at all, along with 14 Axis brand cameras that don't have it either. It is only the WB70A's and the new Vivotek's we just got that seem to be having this issue with corrupt JPEGs.

Attached are stills from the cameras with corrupt picture. The one with "332" is from the Vivotek, and the "1153" is from a Toshiba IK-WB70A.

For the longest time the Toshiba's had this problem but whenever I connected to the cameras webpage, the issue stopped (even before the stream started up). Then after they were all about 2-3 years old, the EXACT same JPEG corruption was appearing in the picture on the cameras webpage itself, so I knew it wasn't ZM fault then. I sent in one of my Toshiba's in a year ago for diagnostics and they couldn't recreate the issue. This Vivotek that is just now starting to experience similar corruption is about 8 months old. The other 7 I just got a few weeks ago. I'm going to be contacting Vivotek to see if I can get some help from them as well of course.

Anyway i've been seeing this across multiple versions of ZM and OS's as you can see, and i've tried tons of things to resolve it. Things like changing resolution, frame rate, stream file name... checking hardware, trying different switches, etc. My DVR network is completely independent with 100mbit POE switches. Nothing goes over WiFi, there isn't even a WiFi device on the network. They are all the same brand and model of switches but i've tried others to see if it was the switches, and it isn't. I've tried different firmware versions on all the cameras as well.

I even tried setting one camera and one of my DVR's second NIC onto a different network segment with its own switch, so it was 100% by itself and the issue still occurred. So it isn't my network being overloaded. Which makes sense when considering the 30+ other model IP cameras on this network aren't having this issue. None of my Axis brand cams have this issue, and none of the Toshiba IK-WD01A's do either.

Another interesting thing that makes me think it is the cameras is it appears the Vivotek's and Toshiba's may be made by the same manufacturer. The installation (camera detection) software is the same and they use the same URL for streaming motion JPEG. I could be wrong but the physical hardware design between the Vivotek and Toshiba camera are also strikingly similar. I initially thought switching to a different brand would help but now it appears my efforts may have been for naught. Regardless, I feel determined to figure out what the heck is going on and if it is indeed the cameras (which everything i've done shows that it is) then I may have to stick to Axis brand cameras only, since I don't have this problem with them. Only problem is they tend to be a bit more expensive.

So is anyone else out there experiencing this issue?

[linuxsense]

What is your zm config for the cam(s) with corrupt images? Did you install from the git source or use a deb?

[gregom]

Thanks for replying. Frame rate is blank, its set at the camera. Everything else is default. I put in the host IP address, port 80, call for /video.jpg, 24 bit color, and the res for the camera.

I installed ZM from the repository, so deb i'm assuming. apt-get -y install zoneminder

[linuxsense]

I am not very familiar with the current state of 1.27 deb packages of ZM as I always build from source as the development of ZM is pretty rapid. I would suggest going with 14.04 and building ZM with from source on a minimal install of 14.04 with the vlc libs from the 'nightly' videolan repo for best results. It can be a pita but once you have all the dependencies in place building newer versions takes minutes. I recently did a fresh install of 1.27.1 on 14.04 x86_64 and I can share my build notes with you if you would like to take a crack at it. The performance of my new build is great and without some issues I had on 13.04. 14.04 has a few gotchas that make it slightly more difficult compared to older Ubuntu releases but I think I documented all of them.

Have you tried pulling RTSP streams from the IP cams using ffmpeg or libvlc?

[gregom]

Well as I stated I get JPEG corruption on my older DVR's as well running Ubuntu 8.04 and ZM 1.23.3. I also have this one new Vivotek camera being recorded on one of my older DVR's, albeit at a lower res. I went and checked and it also saw the corrupt JPEG. So these are two completely different builds, different OS's, different hardware as well and the same issue occurred from the same camera. I honestly think it is the camera, or something causing corrupt data on our network. Yet the issue only occurs with specific models of cameras.

I haven't tried experimenting with RTSP or any other stream capture method... I figured since ZM wants to save JPEG's, why not send it JPEG's? It seemed the easiest and safest way to do it, but maybe that isn't the case anymore. I'll have to look into figuring out how to do that.

I'm certainly open to trying your recommendation for a build, but again I don't think it has anything to do with ZM or how my DVR's are built. I could be wrong.... but I just go with the information I have.

[linuxsense]

The issue inst necessarily zm specific, it can happen for various reasons, but using RTSP/RTP with ffmpeg or libvlc is the way to go if your camera(s) support it. It is just a better way to handle 'real time' streams. Without log info (you might want to crank up the verbosity and try to locate when a corrupt image was captured) its hard to say but your problem could be related to an increase in latency between the cam and server, an overloaded server, or lost packets. Probably a few other things as well With the latest zm code from the git repo, a specific snapshot of ffmpeg (link at bottom), vlc libs from the videolan 'nightly' repo (good as of a few days ago), libjpeg-turbo, and a few other small items I get zero image corruption from any of my HD IP cams or the analog cams. For whatever reason I have a much lower load on the server compared to a similar build on 13.04. Runs great. If you want to give it a go send me a PM and I can share my build notes. They are not a replacement for a 'how to' but it has all the info on which packages to build from source, from Ubuntu repos, and other repos. I believe that the gentleman that provides most of the Ubuntu ZM packages is going to be releasing 1.27.1 packages soon once a few issues are sorted so waiting on that is an option too but I don't know if he is only making 14.04 packages or older ones as well.


Link to the elusive 'known to work well with ZM' ffmpeg snapshot: https://dl.dropboxusercontent.com/u/651 ... 688.tar.gz

[gregom]

Yeah I think all my IP cameras support it. I think I will look into trying it.

I don't really have the time to start testing a new build right now but would love to try it later. If it isn't too much work i'd like your build notes so I can try it at a later time. We are always looking to improve our DVR systems and camera infrastructure. This is actually our first successful new build. We've been running (and still are) on ZM 1.23.3 for a long time now.

[hesral]

I have limited experience with vivotec, but two fisheye cameras I tried had problems keeping up the stream when the resolution got too big. And the "no framerate in zoneminder" thing may not work, if the camera isn't feeding the image data as a stream of continuous data. If it is just a single frame per http get request, then zoneminder have no way to know when to get the next frame, unless you tell it how often it should request a new image. So it will probably just do it as fast as possible. And that may overload the camera. Try setting the framerate in zoneminder to 1, or 5, and see if that helps.

[knight-of-ni]

Try the libvlc method. It was implemented specfically to workaround corrupt video issues when using the ffmpeg method.

[gregom]

Makes sense... I might give that a try.

I was able to get RTSP to work with ffmpeg. Same camera, same res and frame rate, just another stream. Seems to be working fine except a few video artifacts fairly often. I'm going to let it run for a while for testing purposes. This is on another server BTW, same hardware and software.

I'm also recording a few of the Toshiba cameras on RTSP that frequent the correct JPEG issue. I am running both at the same time but on different servers, will see if I get corruption on the RTSP recordings just as with MJPEG.

I tried libvlc but couldn't get that to work. I'll have to do more research as i've never tried MPEG/H.264 streaming before so i'm still learning.

Thanks for the help.

[linuxsense]

If you are still using ffmpeg you might want to add these options under the 'source tab' for the monitor:

Code: Select all

allowed_media_types=video, bitrate=4096

Some of my cams make ffmpeg think they do audio even though they dont, telling it to not look for audio seems to keep things stable. I also use a CBR from the cams so I set it on the ZM side with the 'bitrate' option. Also seems to help a bit. Obviously change 4096 to whatever bitrate you are running, assuming its constant.

Have you tried monitoring the latency between the server and the IP cams that get artifacts? I have seen cams do that when the latency on the LAN spikes up. I wouldn't be surprised if the artifacts appear during a time of inconsistent or high latency.