purgewhenfull removes all events

[howardtopher]

I'm very new to ZM, as in a couple weeks. So far I've been able to work through all issues with searching the forums and the wiki and I have all of my cameras working and in modetect mode. However, I've hit an issue I've been unable to resolve.

I have modified the purgewhenfull filter to kick in when the disk gets to 90% full or higher, which it does just fine. I've also set it to limit to the first 10 events ordered with ascending id order as this should get the 10 oldest events. The goal is to keep as many events on disk as possible while not eating up the entire disk. I'd like to see it hover between 85%-90% at all times. I would expect 10 events per run would keep disk usage near 90% at all times.

With no recorded events, my hard disk is at 49% full (it's a small disk, I'm going to add more after I get ZM running how I'd like). When it gets to 90% and the filter kicks in it wipes them all out back down to 49% full. There are usually a couple hundred events when it gets to that point. I would expect the filter to stop running after the disk usage gets to 89% or lower. I have been completely unable to get it to not do that. Looking at the zmfilter log it does in fact delete all of them during the same second.

Is there something I'm not doing correctly? I'm running ZM on CentOS 6.7 with the 1.28.1 version that comes from the ZM yum repository. I have not built it on my own this go round.

[mikb]

I can confirm there's something you're not doing right, as this is not how "purge when full" should operate -- your assumption of its behaviour is correct, it will trim a few events (in your case, 10) and give up. If disk is still over (nn%) on the next run, it will trim a few again. You may need to increase the number-trimmed-per-run if it fails to keep up, and your disk overflows!

Suggest you post a picture/copy of the filter settings you are actually using and let people see what you've actually got set, which may differ from what you *wanted* to set.

This is not the first time I've seen this reported though ...

[asker]

Please turn on DEBUG and take a look a the query that finally gets formed by zmfilter - it will show in your logs.
https://github.com/ZoneMinder/ZoneMinde ... pl.in#L570

It can often provide you with good insights on what its trying to do. Alternately, just change the line above to Info for now

[howardtopher]

Thanks for the responses. I turned on debugging and restarted zoneminder and never could find the actual query. However, I tried something else and it seems to have the problem resolved. The OPT_FAST_DELETE configuration option was enabled. So even though it was deleting events from the database, it wasn't actually clearing them off of the disk. So the next time the filter ran it deleted 10 more. After turning that off it appears to be good working as expected.

[asker]

Ah. I see - since zm filter is doing a df, even after deleting the events it still kicks in the next sets of deletes as the files don't get deleted till zmaudit is scheduled , potentially removing all your events over multiple cycles.

Out of curiosity that is your audit interval and your filter interval? This could happen only if either the difference in intervals is very large or you have very few events - that is zmfilter might have run an extra few runs before zmaudit cycled in.

Zm filter seems fraught with many such issues and warrants a good code check.

[bbunge]

Turn off (uncheck) OPT_FAST_DELETE. This will delete event images when the database record is deleted. Will take more time and may slow things down a bit but you will not get the wild swings in disk changes. Once you get to a place where things are running well switch to a time based filter. It will usually run at night and remove a days worth of events at a time.

bb