quote:

Originally posted by sigcrazy7:
This is why all my data now lives on file systems that are self-healing. I don’t care to have bits flipping in financial data. Pics are one thing, but small corruption in other files is hard to detect. So I guess it really comes down to how much you value your data. To some people it is irreplaceable, so it being non-commercial is not really relevant.

Very good point. Thanks

quote:

Originally posted by Bytes:

quote:

Originally posted by sigcrazy7:
This is why all my data now lives on file systems that are self-healing. I don’t care to have bits flipping in financial data. Pics are one thing, but small corruption in other files is hard to detect. So I guess it really comes down to how much you value your data. To some people it is irreplaceable, so it being non-commercial is not really relevant.

Very good point. Thanks

Ugh. Can you please tell me about these self healing systems!!

You're moving towards the deep end of the pool.
if you want to understand parity there is a decent wiki via google search. if you want to just gloss over the highlights check https://www.tutorialspoint.com/ under the parity correction section. I think reading the info will do a much better job than me trying to explain it. I am not an it pro, I just enjoy playing with it and learning at my own pace. I tried reading all about it and came to the conclusion that I was way over my head. I now just accept that its a good thing and include it in my system design when possible.

I somewhat touched on it in my first post, when saying you should look at a four bay NAS. Because I have been stung by bitrot as discussed prior, I am mostly sensitive to that. However, btrfs has many other benefits. The Wiki article does a decent job of explaining it.

To save you from going through the btrfs Wiki , here is the part about data scrubbing.

quote:

CRC-32C checksums are computed for both data and metadata and stored as checksum items in a checksum tree. There is room for 256 bits of metadata checksums and up to a full node (roughly 4 KB or more) for data checksums. Btrfs has provisions for additional checksum algorithms to be added in future versions of the file system.[35][85]

There is one checksum item per contiguous run of allocated blocks, with per-block checksums packed end-to-end into the item data. If there are more checksums than can fit, they spill into another checksum item in a new leaf. If the file system detects a checksum mismatch while reading a block, it first tries to obtain (or create) a good copy of this block from another device – if internal mirroring or RAID techniques are in use.[86][87]

Btrfs can initiate an online check of the entire file system by triggering a file system scrub job that is performed in the background. The scrub job scans the entire file system for integrity and automatically attempts to report and repair any bad blocks it finds along the way.[86][88]

I have my NAS scrub my entire system every six months. It does it automatically, in the background, during low usage hours like at night. This happens on my main NAS and on my backup NAS(es). During scrubbing, no services are interrupted. I don't really know that it's happening.

I don't want to give the impression that you must have a parity RAID for the filesystem to be self healing. A b-tree filesystem can fix itself even if it exists on a single disk, but it must use the block parity to do so, FWIU. However, by having a RAID, it gives the filesystem more options. So, say it finds an error in the block through checksum error, it will try to find a different copy of the file first. If you have a RAID 1 system (two disks, mirrored), it will look to the file on the mirrored disk first. If you have a parity RAID (5,6,SHR), it will use the RAID parity to fix the block error. Only if a good copy is unavailable will it then look to snapshots and replication. If those are unavailable, then it will attempt to fix the bit through the block parity.

This is just my understanding. People who actually work on the filesystems can feel free to correct me. My story is like itz400. I don't work on these filesystem, but I use them, so what I know is from my own reading and implementation of these systems. I have been able to recover accidentally deleted files through going to older snapshots and getting the file. I also have continuous backups, but getting the file from a snapshot or a replicated folder is faster.