debugging load average and ZM jpeg saving vs shinob i h264

[marcmerlin]

I've had zoneminder for a long time, upgraded to 1.30 not that long ago (yes, I know, 1.31 just came out it seems), and I can tell my system is stressed by the load, even if it manages still it seems.
I have 12 cameras, most are 1080p. I have them as Mocord, which does mean they are recording at all times.
I have maximum FPS set to 7 and alarm maximum FPS also set to 7
All cameras are set for jpeg or mjpeg capture, so there is no H264 video decoding
Cameras are recording to a double drive raid0 array for a bit of extra speed, on btrfs on top of bcache backed by an ssd.
CPU-wise I have a quad core Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
32GB of RAM
OS is currently 32bit to save on resources (smaller pointers, smaller binaries), but kernel is 64bit

I was curious that shinobi claims
"JPEG Storage is awful
Saving each frame as a separate file in JPEG format can have a seriously detrimental effect on storage space and the hardware itself.
Shinobi saves to WebM and MP4 files. While MP4 takes a fair amount of space.. its level of CPU usage during encoding for H.264 streams is just amazing."

Usually I like benchmarks and data over an empty sentence that includes "speed is amazing" with absolutely no info on the I/O vs CPU tradeoff.
But I'm curious, have others looked into this? Am I better off staying with jpeg frame storage over H264 from shinobi?
Part of my "problem" I guess is that I'm using mocord, and currently have bulk frames set to 100, but this is not working well for me because the whole point of mocord is that I want to record even if no motion is detected, and if ZM detects no motion it collapses a lot of frames together, even though they're not quite the same, and they may have data I wanted to look at (this has happened before sadly and the video frames I wanted were gone because ZM didn't see movement and junked them as part of bulk processing).
If I were recording to video instead, the video codec would record just the difference and let me replay the frames and let me see if there is a difference that's important to me, but not enough to trigger motion detection.

That being said, I'm also getting really weird recording from event recording.
Look below, I have recordings of 22 seconds, and others of 55mn with 9811 frames (!)

Id Name Monitor Cause Time(v) Duration Frames Alarm Frames Total Score Avg. Score Max. Score
4882250 Event-4882250 cam-frontyard Continuous 03/12 15:34:34 00:05:25 2151 0 0 0 0
4882227 Event-4882227 cam-frontyard Continuous 03/12 15:30:00 00:04:36 1677 7 46 6 8
4882198 Event-4882198 cam-frontyard Continuous 03/12 15:20:00 00:09:59 3811 0 0 0 0
4882174 Event-4882174 cam-frontyard Continuous 03/12 15:10:00 00:10:00 3780 0 0 0 0
4882145 Event-4882145 cam-frontyard Continuous 03/12 15:05:38 00:04:21 1556 0 0 0 0
4882069 Event-4882069 cam-frontyard Continuous 03/12 14:37:36 00:27:58 5209 4950 265363 53 60
4882055 Event-4882055 cam-frontyard Continuous 03/12 14:31:44 00:05:53 1842 397 21995 55 59
4882044 Event-4882044 cam-frontyard Continuous 03/12 14:30:00 00:01:45 413 227 13115 57 60
4882039 Event-4882039 cam-frontyard Continuous 03/12 14:29:30 00:00:30 203 0 0 0 0
4882033 Event-4882033 cam-frontyard Continuous 03/12 14:28:37 00:00:52 225 94 5316 56 59
4882030 Event-4882030 cam-frontyard Continuous 03/12 14:27:33 00:01:06 258 121 6999 57 60
4882010 Event-4882010 cam-frontyard Continuous 03/12 14:20:46 00:06:47 1528 1127 61162 54 60
4882003 Event-4882003 cam-frontyard Continuous 03/12 14:20:00 00:00:46 226 4 221 55 56
4882000 Event-4882000 cam-frontyard Continuous 03/12 14:19:54 00:00:05 35 0 0 0 0
4881996 Event-4881996 cam-frontyard Continuous 03/12 14:19:28 00:00:25 138 1 59 59 59
4881992 Event-4881992 cam-frontyard Continuous 03/12 14:18:29 00:00:59 329 30 1736 57 59
4881991 Event-4881991 cam-frontyard Continuous 03/12 14:18:03 00:00:25 123 10 576 57 59
4881989 Event-4881989 cam-frontyard Continuous 03/12 14:16:26 00:01:37 513 58 3311 57 59
4881985 Event-4881985 cam-frontyard Continuous 03/12 14:15:01 00:00:00 1 0 0 0 0
4881986 Event-4881986 cam-frontyard Continuous 03/12 14:15:01 00:01:24 462 26 1473 56 59
4881844 Event-4881844 cam-frontyard Continuous 03/12 13:20:00 00:55:00 9811 8642 409480 47 60
4881826 Event-4881826 cam-frontyard Continuous 03/12 13:17:26 00:02:33 976 0 0 0 0
4881809 Event-4881809 cam-frontyard Continuous 03/12 13:10:00 00:07:12 1672 748 14318 19 30
4881798 Event-4881798 cam-frontyard Continuous 03/12 13:08:13 00:01:46 671 0 0 0 0
4881793 Event-4881793 cam-frontyard Continuous 03/12 13:05:39 00:02:28 596 15 75 5 5

This may be a totally different problem though.
Anyway, back to recording and jpeg recording vs shinobi h264, what are your thoughts?

[bbunge]

First off you have two settings that should be blank unless you have local cameras: Maximum FPS and Alarm Maximum FPS
With streaming cameras (AKA IP cameras) set the frame rate at each camera as well as the resolution. My recommendation is 5 FPS with 640x480.
1.31.x is still in beta testing and not officially released.
Performance of your system depends on a lot of factors: LAN quality, camera output, server processor and hard drive speed. I'm running 17 IP cameras at 640x480 5 FPS on an i5 Quad Core (3.2 GHZ), 16 GIG RAM and a WD Blue 1 TB desktop drive. Four cameras on Mocord during the day with at least 5 PC's running Montage of the cameras. The system is not stressed at all and I'm considering reducing the RAM to 8 GIG.
I've recently set up a used Poweredge server with RAID 1 on 250 GIG drives for the OS and a 1 TB drive for events (see: https://wiki.zoneminder.com/Common_Issu ... ive_or_NAS). This "old dog" is fast in testing compared to a single drive system and may get upgraded to Ubuntu 18.04 before it is put in service.
But I've dodged the issue of .jpg files vs. video. And that like all things depends. Video, like movie film, is still created from a series of still pictures. So it boils down to where you want to use processing power. A security camera system is not a HD movie machine but a balance of frame rate and image size to catch that thieving rodent!

[marcmerlin]

Thanks bbunge. I do jpeg pulls, isn't that speed controlled by my server? (I don't like H264 pushed by the camera because if my PC hangs for a bit, or the network is congested, the H264 stream gets damaged and I get broken frames. With jpeg pulls, I know my PC controls the data pull, and doesn't have to waste CPU in H264 decoding).

As for doing 640x480, not really, I do at least 1920x1080 for my lowest res cams, some are a bit higher res. Yes, that's expensive, but if I can't see who is trying to steal from my mailbox or get their license plate, let's say, what's the point of the camera?
For your system doing ok, you' have 4 to 6x less load per camera than I do due to the lower res, so I'm not surprised yours is doing fine.
As saving output as H264 video instead of jpeg frames, it sure would save me disk space if I turned down bulk from 100 to 0, but I'm indeed worried about the CPU impact and don't quite trust shinob i's claim that "speed is amazing".
I'm kind of curious if people have tried both and can report the results.

[knight-of-ni]

Hey, I like the way you think. I don't have to tell you that, the author of the software of which you speak, loves to make inaccurate statements, just like the one you quoted, to promote himself. Those of us with real experience in the industry know that jpegs are still quite useful, especially since storage is so cheap. In any case, you are asking the right questions. Ignore the hype. Rather, collect real data. Then make your own decision on what product you want to use.

Along the lines of what bbunge said, you've told us what values you've got in the max fps fields, but the important thing to know is the framerate of the stream itself coming from the camera. If the framerate in the camera is higher than 7, then you aren't saving cpu resources at all. You are increasing the cpu load.

If you want to stick to the latest release of zoneminder, the way to minimize cpu usage is:
- set the frame rate in the camera to something low. I like to use 5 fps, and yes that is plenty, even for moving objects. bbunge likes to go lower.
- don't use the max fps fields (This is a bug that awaits someone with time to fix it. You *can* put a value in here higher than the camera's framerate. )
- set your camera streams to mjpeg rather than h264. Cpu usage will drop like a rock, at the expense of significantly more bandwidth.
- Lowering the resolution will significantly lessen cpu load, but I totally get it if you want to keep that 1080p image. I would too.

If you want to live on the edge, try one of our development packages:
- set the framerate in the camera to something low
- keep max fps fields blank
- enable h264 passthrough in the monitor config, which will save straight to mp4
- disable jpeg recording

I saw your other thread about bulk frames. Despite having used ZoneMinder for a decade, there are still some features I've just never used, and that is one of them. It will be interesting to see if you get any responses. Generally speaking however, the things that use the most cpu resources are resolution, framerate, compression type. Motion detection not so much these days, and I would not expect bulk frames to matter much either.

Lastly, the author if which you speak loves to masquerade under different names. If a post shows up in this thread from a new user, trash talking zoneminder or those that participate in the project, I'll give you one guess who that is.

[marcmerlin]

Thanks for the reply. Yes, I smelled BS there, glad to hear confirmation
But you are both talking about framerate coming from the camera. I'm doing jpeg http pulls from the camera. It's not pushing anything to me, I'm pulling from it.
I agree that even 3fps is enough actually, I'm just not even getting that most of the time, because I have a bottleneck somewhere I haven't found yet.
But you're saying I should use mjpeg push, sadly many of my cameras do it badly or not at all. Is it really better than jpeg pull? I thought that outside of the http round trip for each jpeg pull, it'd be about the same?
Thanks also on the warning about the other software's author

[knight-of-ni]

Ah, I see. We call that snapshot mode. I misunderstood that was what you were doing, so yes you certainly can use the max fps fields.

I don't know for certain how much better mjpeg would be than jpeg snapshots.
It would be worth it to test which method is more cpu efficient. Watch your zma & zmc processes of a particular camera in snapshot mode to get a rough idea of the cpu utilization. Then change over to an mjpeg stream and repeat the process. That is of course if the camera in question supports mjpeg.

All other other things being equal, I am reasonably certain the mjpeg stream will be more efficient, but I am unsure by how much. It comes down to just how expensive it is to constantly open & close a connection to the camera, which is what is going on in snapshot mode, as you have noted.

I see what you mean from your first post about unexpected event statistics. It's really hard to know what caused some of the more unusual looking statistics without knowing a lot more about what was going on with the hardware, the motion zone configuration, and what was in the camera's field of view at the time the event occurred. I am curious to know what drove you to choose this mode. Would you be willing to use normal modect with a super sensitive motion zone? It's typically easy to make the zone very sensitive so it records even the slightest movement. I don't know at this point if disk io is your bottleneck, but switching over to record only on motion would certainly help with that.
Speaking of bottlenecks, zoneminder will leave a warning in your logs when it can't keep up and your buffers are about to be overrun. Do you see anything like that?

[marcmerlin]

thanks for mjpeg vs jpeg pull. I'm pretty sure it lowers my frame rate a bit, but doesn't otherwise affect me much, so I'm not too worried there.
Why mocord? If I use modetect and have a super sensitive zone, then I'll have even more data written down to disk since bulk won't work, and I'll have to tweak my job that deletes 0 score frames after x days intead of y days (y>x)
Mar 14 22:05:55 gargamel zma_m2[23240]: INF [In overload mode, 2 frames of 3 remaining]
I only have a single camera overloaded, and unsurprisingly it's the only one I can't do jpeg pulls from and that is pushing h264 to me.

For the rest, I have:
Mar 14 22:07:57 gargamel zmc_m2[11336]: INF [cam-fmr: 879000 - Capturing at 5.08 fps]
Mar 14 22:08:16 gargamel zma_m5[22542]: INF [cam-fmr-kitch: 787000 - Analysing at 4.95 fps]
Mar 14 22:08:19 gargamel zma_m10[31832]: INF [cam-kitchen: 1458000 - Analysing at 9.62 fps]
Mar 14 22:08:33 gargamel zmc_m4[11337]: INF [cam-garage: 865000 - Capturing at 5.00 fps]
Mar 14 22:08:34 gargamel zmc_m10[11334]: INF [cam-kitchen: 1653000 - Capturing at 9.90 fps]
Mar 14 22:08:36 gargamel zmc_m24[26146]: INF [cam-mailbox-front: 10000 - Capturing at 1.95 fps]
Mar 14 22:08:42 gargamel zma_m11[24888]: INF [cam-lvr: 1327000 - Analysing at 9.71 fps]
Mar 14 22:08:47 gargamel zma_m4[4446]: INF [cam-garage: 723000 - Analysing at 4.98 fps]
Mar 14 22:09:03 gargamel zmc_m3[11352]: INF [cam-frontdoor: 812000 - Capturing at 4.85 fps]
Mar 14 22:09:04 gargamel zmc_m11[11342]: INF [cam-lvr: 1676000 - Capturing at 10.00 fps]
Mar 14 22:09:05 gargamel zma_m2[23240]: INF [cam-fmr: 752000 - Analysing at 5.03 fps]
Mar 14 22:09:44 gargamel zmc_m6[11323]: INF [cam1-garage: 115000 - Capturing at 0.70 fps]
Mar 14 22:10:00 gargamel zma_m10[31832]: INF [cam-kitchen: 1459000 - Analysing at 9.90 fps]
Mar 14 22:10:12 gargamel zma_m16[22954]: INF [cam-backyard: 613000 - Analysing at 4.55 fps]
Mar 14 22:10:14 gargamel zma_m6[12123]: INF [cam1-garage: 115000 - Analysing at 0.70 fps]
Mar 14 22:10:14 gargamel zmc_m10[11334]: INF [cam-kitchen: 1654000 - Capturing at 10.00 fps]
Mar 14 22:10:23 gargamel zma_m11[24888]: INF [cam-lvr: 1328000 - Analysing at 9.90 fps]
Mar 14 22:10:29 gargamel zmc_m17[25716]: INF [cam-mailbox-rear2: 303000 - Capturing at 6.02 fps]
Mar 14 22:10:45 gargamel zmc_m16[11330]: INF [cam-backyard: 684000 - Capturing at 4.57 fps]
Mar 14 22:10:45 gargamel zmc_m11[11342]: INF [cam-lvr: 1677000 - Capturing at 9.90 fps]
Mar 14 22:10:46 gargamel zmc_m5[11354]: INF [cam-fmr-kitch: 863000 - Capturing at 4.95 fps]
Mar 14 22:11:14 gargamel zmc_m2[11336]: INF [cam-fmr: 880000 - Capturing at 5.08 fps]
Mar 14 22:11:32 gargamel zma_m23[27367]: INF [cam-mailbox-rear: 303000 - Analysing at 1.92 fps]
Mar 14 22:11:37 gargamel zma_m5[22542]: INF [cam-fmr-kitch: 788000 - Analysing at 4.98 fps]
Mar 14 22:11:41 gargamel zma_m10[31832]: INF [cam-kitchen: 1460000 - Analysing at 9.90 fps]
Mar 14 22:11:54 gargamel zmc_m4[11337]: INF [cam-garage: 866000 - Capturing at 4.98 fps]
Mar 14 22:11:55 gargamel zmc_m10[11334]: INF [cam-kitchen: 1655000 - Capturing at 9.90 fps]

Interesting that I don't get the same interval of logging for all cameras, but oh well...
Anyway, the logs look ok enough, but in real life I often get multi second gaps between frames. I need to go back to see if it's only in non alarm state or in all states...

[rockedge]

I have tested both sets of software extensively... I found ZM to be more reliable in the long run especially with motion detection. ZM can be modified and "suped up" and being able to interact with ZM using PERL, PYTHON, PHP or shell scripts put it into a different league.
Overall system loads in my experience and general wear and tear on the machine hardware seem less with ZM than the software who's name we don't mention. The potential to use the jpegs for object recognition analysis is much greater than trying to do the same using the streams produced by Shinob i. Just my experience.

[marcmerlin]

Thanks rockedge, this is very useful feedback. I very much appreciate that you're saving me the time to try the other software to see whether it's faster on my system, or not.

[knight-of-ni]

Excellent objective feedback from Rockedge. Thank you.

[endtyranny]

I am guessing the end result is here? viewtopic.php?f=40&t=29286#p116356

[marcmerlin]

Kudos on your memory and piecing the 2 posts together.

Indeed. honestly I stuck with ZM so much longer than I should have
1) ZM deals badly with 15 cameras in the 2Mpix range, by badly I mean that even with an SSD it cannot deal with the frames fast enough and would drop lots of data, including around the time thieves came to raid our mailboxes. 3 times this happened, 3 times most of my cameras failed to record useful data (and I was in mocord). ZM just didn't keep up, and it's not wireless, I had gigie wired ethernet two my cams,. However my 8 core system was over taxed by ZM (never mind all the RAM I had to add to keep up, but that wasn't enough).

2) generally the amount of traffic and storage from jpeg pulls or decoding h264 streams to jpegs, was not acceptable. Shinob i does the right thing by recording h264 untouched (no CPU used) and decoding in low res to do motion detection

3) obviously the fact that on top of working so badly, a ZM upgrade disabled half my cameras forever without any real clue and anyone being able to tell me why, didn't help. Upgrading to 1.32 broke me, 1.33 didn't fix it, 1.34 didn't fix it.

4) any project that releases non working releases and for which the answer is 'oh, you should run git master' is a huge clue to run. That's always been true in my experience, regardless of the project.

5) and generally the fact that ZM does ridiculous things like trying to ban the word shinob i , should be another clue to run. If the project is so insecure about competition, that's not a good sign.

I'm just sorry I didn't run earlier when I got the first clues.
Sorry that I have to be negative, I'm just upset that I didn't switch earlier., and now I can use my 5Mpix cameras without any fear (I didn't even dare unpacking them with ZM).

Oh yeah, I almost forgot the bug tracker that closes your bugs every few months without anyone ever looking at them, and you just keep re-opening forever, while it keeps reclosing forever.

Now, I'll be honest that shinob i took me a few days to setup, but the discord channel was pretty helpful in getting me almost real time help to get up and running.

[marcmerlin]

Oh, one thing I did forget: shinob i does not deal well with jpeg/mjpeg only cameras. If you have those, stick with ZM, but then again, maybe it was a clue to me that those cameras are 640x480, they're over 10 years old, and maybe I can afford some replacement H264 1080p camera for $30 a piece

[asker]

Indeed. honestly I stuck with ZM so much longer than I should have

So we've heard. Why continue to use this forum to vent if you have moved on?

1) ZM deals badly with 15 cameras in the 2Mpix range, by badly I mean that even with an SSD it cannot deal with the frames fast enough and would drop lots of data, including around the time thieves came to raid our mailboxes. 3 times this happened, 3 times most of my cameras failed to record useful data (and I was in mocord). ZM just didn't keep up, and it's not wireless, I had gigie wired ethernet two my cams,. However my 8 core system was over taxed by ZM (never mind all the RAM I had to add to keep up, but that wasn't enough).

It seems to me your install/fine tuning may also have had issues. I am using a system that has over 100+ cameras to test(I don't host it). In my house, I use a 9 camera system, each 3MP. ZM more than keeps up, 3 years running, on a desktop 32GB, 4core machine.

2) generally the amount of traffic and storage from jpeg pulls or decoding h264 streams to jpegs, was not acceptable. Shhyinob i does the right thing by recording h264 untouched (no CPU used) and decoding in low res to do motion detection

That I agree with - I think ZM needs a pure passthrough option.

3) obviously the fact that on top of working so badly, a ZM upgrade disabled half my cameras forever without any real clue and anyone being able to tell me why, didn't help. Upgrading to 1.32 broke me, 1.33 didn't fix it, 1.34 didn't fix it.

Nothing obvious about it. If what you are saying is true, it would affect everyone. I discovered ZM when it was 1.29 and upgraded pretty much to every release. I've _never_ had a situation where everything broke. I did have such situations in certain breaking releases, like when they moved the location of cgi-bin and once when they did CSS cache busting. Both times, it was documented but I chose not to read the release notes and follow instructions. Of course, there have been several times when "something" broke - but it was never a complete meltdown. Don't expect people to tell you why. ZM's support isn't great. You'll have to figure it out or join slack and ask. If that doesn't work for you, you need to migrate to another system that hand holds you more, which you might have done. That by no means implies ZM is terrible. It means ZM and you are not an ideal pairing.

4) any project that releases non working releases and for which the answer is 'oh, you should run git master' is a huge clue to run. That's always been true in my experience, regardless of the project.

Not true. There have been bugs in releases, but I don't recall a single release that was "non working" (as in "dead on install"). That is true with most open source.

5) and generally the fact that ZM does ridiculous things like trying to ban the word shinob i , should be another clue to run. If the project is so insecure about competition, that's not a good sign.

Just about at this stage, you are sounding like a Shin*bi advertiser. Do you know why that word was banned? At least do some research first. The author was a massive spammer who used these forums to send PMs to people to ask them to try his software.
He frequently fakes his name as different user ids and posts negative things in this forum. If that software was so amazing, I'm sure there wouldn't be a need to spam other forums about it? If you want to be taken seriously, at least act principled.

Don't believe me? Take a look again at who bumped this post up after 2 yrs. "endtyranny" is the author of Shin*bi under a fake name. I just did a whois on that IP and saw that person's email. Look at his posting history as well. Classy, huh?

I'm just sorry I didn't run earlier when I got the first clues.
Sorry that I have to be negative, I'm just upset that I didn't switch earlier., and now I can use my 5Mpix cameras without any fear (I didn't even dare unpacking them with ZM).

I don't think you are sorry. I don't think you ran either. You still seem to be here. You'll find several folks here receiving help. And solving problems. And using ZM. It is always possible that someone doesn't like some software. So move on, completely.

Oh yeah, I almost forgot the bug tracker that closes your bugs every few months without anyone ever looking at them, and you just keep re-opening forever, while it keeps reclosing forever.

ZM's support isn't good in my opinion. It's mostly a 1 member team. The general expectation is you know what you are doing and are able to problem solve. GH issues are meant for bug reports, not "I did something wrong, so it is a bug" - those often get closed. Unfortunately, it also isn't meant for "I need help". I wish it were, but there aren't enough volunteers to help out. May not work for you.

[marcmerlin]

This is tricky. I'm not going to the board anymore, but I get an Email when there is a reply. Should I ignore it?
Once my curiosity got the better of me and I read it, should I pretend that I didn't ?
Would you rather not know that you didn't type this message for nothing and I did read it?

I'm glad to read that I got unlucky in my experience. Obviously I'm a upset at all the time I spent making ZM work, how it slowly started working less over time, as a result of me putting too many cameras, and from what you said, maybe doing it the wrong way (I mostly did jpeg pulls because I had too many issues with H264 desync causing broken frames and fake events on older cameras, and the amount of jpegs being processed, somehow did not make it to my SSD in time. Googling shows other people experiencing CPU issues with ZM as they added cameras with higher resolutions). That said, being upset did not give me the right to be negative, that was wrong, and I apologize for that.

To be fair, shinob will also behave terribly if you configure the cameras wrong (and y that, I mean as terribly as taking my entire system down by lauching ffmpegs quicker than the other ones finished until my system fell over), I've also learned the hard way that trying to use jpeg or mpeg cameras with shinob is a exercise in frustration. It would use 100% of a core per camera if I did that and eventually cause my 32GB/8core system to grind to a halt (ZM instead failed record data but never once took my machine down). H264 passthrough works perfectly, but anything else did not in my setup. If I had learned/known more about newer features in ZM, maybe I could have configured it to use less load.

As for why it broke for me, I guess my install was special or too old, can't say. I didn't do anything other than installing the official debian packages (and yes, I had the cgi-bin and CSS problems too, looked them up and fixed them). And fair enough it was apparent'y only "not working" for me. That's a relief actually as I don't wish to anyone the time I went through to setup 2 different systems and the time cost of switching.
Shinob made debugging cameras easier for me, but so far I've found that setting up and debugging motion, has been less obvious. ZM was easier.

As for the spam option you refer to, sorry, I didn't know the details, thanks for explaining. Yeah, I agree this sucks, but at the same time, I don't think almost banning discussions and comparisons is helpful either. Ideally both projects could benefit from one another even if obviously given the entirely different codebase and language, they'll never merge.
For the rest, your points on my being upset and negative are fair, I have the right to be upset after all the time I lost with problem, thieves that were supposed to be caught on camera, but were not, and then my whole setup going down for good after an upgrade I could not revert, but not the one to be negative, and obviously I'm only owed a full refund of the $0 I paid for the software . My apologies for that.

Thanks for taking the time to answer, now it's pretty clear that my experience was not typical, and that's good to know.

Cheers.