ZoneMinder Deletes 120GB+ (All?) Events With Little Provocation: Logs Inaccessable

[mysterylectricity]

I searched through old discussions and found some near misses but lost my patience, hope you don't mind my asking directly:

Setup is ZM 1.30.4 under Ubuntu Server 17.10, 250MB SSD, Logitech C920 webcams (which have good low-light sensitivity for the price, BTW). Remote control via Chrome, though also tested against Edge: said remote system has retrograde Java V7 due to limitations imposed by Ubiquiti mFi , which I would have no truck with if I had not found a large cache of same for some few $ at the Salvation Army Store.

Everything was humming along fine for many months. ZM was hogging disk space leaving only about %5 free, but that was both entirely expected and entirely acceptable. The machine is basically dedicated to ZM.

At one point about 3 weeks ago I noticed I was losing significant events. I started fiddling with blending and alarmed-pixel values, etc. from a remote system (my laptop) on the same subnet.

Now here is where things get interesting. As best I recall, the web interface to ZM froze. So I ssh'd into the ZM host and rebooted.

It was only some days later I noticed that "disk usage" had fallen from about %95 to about %38.

Since then, disk usage has been creeping up as I should expect. Currently %82 and climbing.

Perhaps worth noting is that I can't access the "Log" files from remote host as easily as historically. I did catch a brief glimpse of the log file while composing (ie: fact checking) this post, but I can't seem to duplicate my efforts.

OK, well even I am confounding myself. It seems to me there are two very different issues here:

1) Why did ZM delete so many if not all events across a reboot?
2) Why am I suddenly having so much difficulty accessing log files? Could it be because I downgraded Java? Indeed, does ZM rely on Java?

Than you for any insight.

[rockedge]

I do not think Java is a major component of ZM. This scenario sounds similar to another thread concerning unexpected event deletion. What is the status of the Filters your ZM system is using? What filters are running in the background? Is this what you meant by

At one point about 3 weeks ago I noticed I was losing significant events.

that the events are being created then unexpectedly being deleted? Is the mysql/mariadb Time synced with the system and PHP/Apache time? The filters could go rogue if the Timestamp is incorrect or mis-matched between the database and event paths. Maybe in this case the zmaudit.pl encountered a mismatch and determined the events were orphaned and removed them.

thinking about it...the logs viewer can get weird if there is a time not being in sync issue.

I am just throwing out some ideas........

[bbunge]

With 5% free disk space that means you are using the default settings for the purge filters (95%). I have found that 95% may be too high especially of you are using high frame rate and large resolution. When it hits 95% full disk ZM will start to delete records from the database at 100 per run. zmaudit runs every 5 minutes and will delete the images for those 100 deleted events. It can happen that things get backed up and the disk fills up to the point where Linux chokes. I have had this happen several times on early ZM installs and have had to manually delete images to get it working again. Also, if the images are not deleted the filter will run again and again deleting 100 records at a time. Good way to loose a lot of events.
Your database could need attention. Run mysqltuner to see if paramaters need to be adjusted. innodb_buffer_pool_size = may need to be increased to account for all those events.
You might want to go back to Ubuntu 16.04. Better for the long run than a development version.

[mysterylectricity]

Rockedge I see your point, my wording was ambiguous. What I meant was that significant events in the real world were not being alarmed. For example, while my camera was aimed at my car, someone opened the door to my car (which evidence suggests I not only left unlocked but slightly ajar) and the event was not alarmed.

I dug deeper into the docs and default settings, and noticed that (by default?) zone alarm "Units" was set to "Percentage", and a rather large one at that.

Things went MUCH better when I clicked the switch to alarm on some absolute number of pixels.

However in retrospect I think I may have overcompensated, setting the values for Min/Max alarmed pixels and blend percentages too small, which may have resulted in virtually continuous alarms for a day or two.

Which brings us to bbunge's insight. It's possible that at some point I had ~100 events of ~1GB each, or simply a few events of 10's of GB's, and these got pruned in one fell swoop.

Well, there's still the task of figuring out why I can't access the logs anymore, I'll probably fiddle around with it a bit and then reinitialize the database config file.

Thank you for the insight thus far!

[marcopete87]

Using an SSD for video recording is an BAD idea: you will kill because too much erase-write cycles; don't mind TBW value, it's only an optimistic value: where i work, i replaced some ssd which became paperweight before reaching their rated TBW value (in best scenario, ssd became read only, so i needed only to make an image from disk and write in a new disk, in other scenario, everything was garbage so i needed to configure a new workstation).

Can you check disk for filesystem corruption?

[kingofkya]

Yeah, as someone who worked in a data center i can say even Raid ssds constantly needed changing out on heavy db server because a write endurance issues. And even this its high recommended to have a secondary server for uptime reasons its not uncommon to lose 2 ssds at the same time. Some older ones even drooped to sub kbps range of speeds when near full. I would highly expect that your db is corrupted or partial corrupted and events were never writen to the ssd due to a disk timeout.

Its not the worst thing to have mysql sql on ssd but also putting the video files on it is not a good idea, I would high suspect the ssd is worn out as most consumer disks do not have much wear endurance. As a general rule never run a ssd full it drastically reduces the drive controllers options for wear leveling and your force it to make bad calls and reuses heavily worn flash locations. (a ssds picks a location based on what has lowest use and is currently free) Unless your runing one with a lot of extra flash built in to compensate see sandisk ulta2 or better / intel SSDs. There a few others but thous are ones I commonly use.

I would recommend you get yourself a 500gb+ WD black and move your events on to it, as well as repair you mysql db i am sure its corrupted even if the ssds hasen't fully failed. https://dev.mysql.com/doc/refman/5.7/en ... ables.html

[mysterylectricity]

Yes I'm looking forward to migrating off of SSD to HD's. Originally I was running off a 120 GB flash pen drive (one of those fast ones with an SSD controller/USB bridge internally). Sure enough it more or less locked up on me. I can't remember how long it lasted, somewhere between 6 mos. and one year