File System Recommendation

[Ruler]

I had an old system with 4 x 400 gig IDE drives in a RAID with a reiserfs over top of it for storage. Then the motherboard blew up, probably due to our running 18 cameras on a system that had enough power to run about 4. Got a newer system and installed Debian on a boot drive with a 2T hard drive as storage, mounting it as /var/cache/zoneminder using ext4 as the file system.

Unfortunately, someone vandalized our building over this past weekend and when I went to look at the camera pointing right at where it happened, found that it hasn't been saving anything since January 1. The ZoneMinder logs *insist* that the file system was full, even though df showed over 500 gig available yet. On a hunch, I checked the inodes with df -i & sure enough - the inode table was full.

My question is simple - with reiserfs not having been worked on for well over a decade & scheduled for removal from the kernel and reiser4 incorporation meeting resistance due to Hans Reiser's.... uhhhhmmmmm..... 'legal issues', what file system type is recommended to use on ZoneMinder systems that store such a massive number of files? My preliminary research suggests XFS, but I've never used it & do not want to get bit in the rear end again due to my ignorance like I did with ext4.

---

On a semi-related note, if I stop ZoneMinder, archive the files from /var/cache/zoneminder somewhere, umount the 2T drive, create a new type of file system, mount this new file system under the same mount point, then restore the files from the archive, ZoneMinder should start without losing data or requiring any type of reinstallation or reconfiguration, correct? Is there a configuration backup utility that I can use to download the present configuration, just in case something unexpected happens? (I ask this last because setting up all the cameras in the system, each with specific settings, was a pain that I'd rather avoid if possible.)

[mikb]

Your file system being "full" due to lack of inodes is certainly a real thing (and confusing, if you ever come across it).

I've never tampered with the inode-count settings, but a quick man mkfs.ext3 reminds me that you can set this when creating the filesystem. Not sure you can update it/tune it later though ...

Code: Select all

-N number-of-inodes
              Overrides the default calculation of the number of inodes that should be reserved for the file  system  (which  is
              based  on  the number of blocks and the bytes-per-inode ratio).  This allows the user to specify the number of de‐
              sired inodes directly.

I've always just used the defaults and got away with it, so never had cause to adjust the FS to say -- "no! there will be squillions of tiny files" vs. "there will be hundreds of data-slabs of gigabytes each."

Are you storing individual JPEGs per event, or one video per event? JPGs will eat up inodes faster, 1 inode per frame, rather than 1 inode per event. That may be one way to get it back under control without having to re-make filesystems.

[Ruler]

I've never tampered with the inode-count settings, but a quick man mkfs.ext3 reminds me that you can set this when creating the filesystem. Not sure you can update it/tune it later though ...

Saw that option & found documentation about it that unfortunately states it is not something tweakable without destroying & re-creating the entire file system. If I do that, I'm thinking it might be better to change to a more robust file system that wouldn't suffer from such a limitation while I'm at it. Frankly, I don't even know what I should pick for a number of inodes.... anything I choose would be just a guess at best.

Are you storing individual JPEGs per event, or one video per event? JPGs will eat up inodes faster, 1 inode per frame, rather than 1 inode per event. That may be one way to get it back under control without having to re-make filesystems.

Saving individual frames. Reason is that both of the capture cards are PCI & I needed to find a machine with 2 PCI slots... not the easiest to do & it ended up being an older used machine. Don't know if it would have enough horsepower to build a video for all the cameras.

[Quantum]

XFS

XFS

or

XFS.

[Ruler]

XFS

XFS

or

XFS.

Anything I should be aware of or do while creating the file system? Have you actually used XFS for ZoneMinder? My research indicates that it's geared towards having larger files & suffers when there are many small files being saved. (And hey - I read it on the internet, so it must be true, right? ) Like I said earlier, I've never used XFS before...

[Quantum]

I've used XFS for years (and in the past used Reiser), and XFS is painless and journal-safe. I have five disks which I pass through to the cameras VM then LUKS encrypt, divide into partitions, and mkfs.xfs. I also use XFS in my backups server (over LUKS). I use it whenever given the choice.

[iconnor]

Nothing wrong with plain old ext4

I would not recommend btrfs for video storage... just doesn't fit the usecase and you will find 1 processor pegged on a btrfs process.

[Quantum]

Evidently there is...

BTRFS is mandatory for the OS I use. (MicroOS) I know it's had its problems and I have experienced them all, but its characteristics are ideal for an immutable OS like MicroOS, and it is my experience that 'ButterFS' has matured sufficiently at this point. I've converted my herd of VMs to MicroOS. An immutable OS takes some adjustment, but it's worth it for a security-ingrained type like me.

For those filesystems over which I have a say-so, it's XFS. For disk arrays I've converted from ZFS to BTRFS.

[Ralphred]

Your file system being "full" due to lack of inodes is certainly a real thing

This is what makes me baulk at the idea of using XFS on a filesystem that is storing many small files, and not a few large ones. Ironically resierfs was very good at this. XFS recovery tools are not as good as EXT's in my experience, but that's kind of moot for this sort of storage.
Just up your inode count when you re-format with -i, or alter /etc/mke2fs.conf and half the default to be safe (read the man page for mke2fs under the -T option to find out what "default" would have been used on a filesystem of your size).

[Ruler]

Is there a cap on how many files XFS can store? Research indicates 2^64 - 1 as the limit, which I would think more than sufficient. Given my inexperience with the file system type, I don't know if there's anything I need to set when creating the file system to achieve this or if it's automatic.


If I were to stay with ext4 as the file system, blowing it away and recreating it with an increased setting for the inode table, how would I go about doing this??? The man pages indicate the following:

-i bytes-per-inode
Specify the bytes/inode ratio. mke2fs creates an inode for every bytes-per-inode bytes of space on the disk. The larger the bytes-per-inode ratio, the fewer inodes will be created. This value generally shouldn't be smaller than the blocksize of the filesystem, since in that case more inodes would be made than can ever be used. Be warned that it is not possible to expand the number of inodes on a filesystem after it is created, so be careful deciding the correct value for this parameter.
-I inode-size
Specify the size of each inode in bytes. mke2fs creates 256-byte inodes by default. In kernels after 2.6.10 and some earlier vendor kernels it is possible to utilize inodes larger than 128 bytes to store extended attributes for improved performance. The inode-size value must be a power of 2 larger or equal to 128. The larger the inode-size the more space the inode table will consume, and this reduces the usable space in the filesystem and can also negatively impact performance. Extended attributes stored in large inodes are not visible with older kernels, and such filesystems will not be mountable with 2.4 kernels at all. It is not possible to change this value after the filesystem is created.
-N number-of-inodes
Overrides the default calculation of the number of inodes that should be reserved for the filesystem (which is based on the number of blocks and the bytes-per-inode ratio). This allows the user to specify the number of desired inodes directly.

The file system currently has 1,921,724,608 blocks on it (2T) with an inode table size of 122,101,760. Given that there was roughly 700 gig free when the inode table filled up, I was thinking maybe increase this by 2.5x, or to 305,254,400, to be safe and provide a cushion. The -N option would permit this, but should I pass anything to -i or -I in order to make it more efficient for this case?