I admire the courage to store data on refurbished Seagate hard drives. I prefer SSD storage with some backups using cloud cold storage, because I’m not the one replacing the failing hard drives.
I would also prefer having a large number of high capacity SSDs so I could replace my spinning hard drives.
But even the cheapest high capacity SSD deals are still a lot more expensive than hard drive array.
I’ll continue replacing failing hard drives for a few more years. For me that has meant zero replacements over a decade, though I planned for a 5% annual failure rate and have a spare drive in the case ready to go. I could replace a failed drive from the array in the time takes to shut down, swap a cable to the spare drive, and boot up again.
SSDs also need to be examined for power loss protection. The results with consumer drives are mixed and it’s hard to find good info about how common drives behave. Getting enterprise grade drives with guaranteed PLP from large on-onboard capacitors is ideal, but those are expensive. Spinning hard drives have the benefit of using their rotational inertia to power the drive long enough to finish outstanding writes.
This is going to be a huge anecdote but all the consumer SSD I've had has been dramatically less reliable than HDDs. I've gone through dozens of little SATA and M2 drives and almost every single one of them has failed when put into any kind of server workload. However most of the HDDs I have from the last 10 years are still going strong despite sitting in my NAS and spinning that entire time.
After going deep on the spec sheets and realizing that all but the best consumer drives have miserably low DWPD numbers I switched to enterprise (U.2 style) two years ago. I slam them with logs, metrics data, backups, frequent writes and data transfers, and have had 0 failures.
You can find cheap used enterprise SSDs on ebay. But the problem is that even the most power efficient enterprise SSD (SATA) idle at like 1w. And given the smaller capacities, you need many more to match a hard drive. In the end HDD might actually consume less power than an all flash array + controllers if you need a large capacity.
Used SSDs, especially enterprise ones, are a really bad idea unless you get some really old SLC parts. Flash wears out in a very obvious way that HDDs don't, and keep in mind that enterprise-rated SSDs are deliberately rated to sacrifice retention for endurance.
Agree on SSD for cold storage, that's not a good idea. But you would be surprised by how little used are typical used enterprise SSDs on ebay. This article matches my experience:
Endurance and retention are inversely correlated, and as I mentioned in my original comment, enterprise DC drives are designed to advertise the former at the expense of the latter. The industry standard used to be 5 years retention for consumer and 3 months for enterprise, after reaching the specified TBW. The wear level SMART counter reflects that; "96% remaining" on an enterprise drive may be 40% or less on a consumer one having written the same amount, since the latter is specified to hold the data for longer once its rating has been reached.
Retention is offline retention. Not online. So not sure what point you are trying to make. If it is that SSDs shouldn't be used for cold storage, yeah I agree, and enterprise SSds aren't designed for cold storage. But you seem to be linking retention to TBW, which are largely orthogonal metrics. If you are going to use the SSDs in a NAS, which by definition are running all the time, why would you even care about the rentention rating?
Curious, what's the use case for wanting your data backed-up without fail? Is it personal archives or otherwise (business) archive related?
Not to say you shouldn't backup your data, but personally I wouldn't be to affected if one of my personal drives errored out, especially if they contained unused personal files from 10+ years ago (legal/tax/financials are another matter).
Any data I created, paid to license, or put in significant work to gather has to be backed-up with 3-2-1 rule. Stuff I can download or otherwise obtain again is best effort but not mandatory backup.
Mainly I don't want to lose anything that took work to make or get. Personal photos, videos, source code, documents, and correspondence are the highest priority.
Don't RAID these days. Software won rather drastically, likely because CPUs are finally powerful enough to run all those calculations without much of a hassle.
Software solutions like Windows Storage Spaces, ZFS, XFS, unRAID, etc. etc are "just better" than traditional RAID.
Yes, focus on 2x parity drive solutions, such as ZFS's "raidz2", or other such "equivalent to RAID6" systems. But just focus on software solutions that more easily allow you to move hard drives around without tying them to motherboard-slots or other such hardware issues.
And 'softRAID', like what is on for free on Intel motherboards or AMD Motherboards suck and should be avoided.
------
The best advice I can give is to use a real solution like ZFS, Storage Spaces and the like.
It's not sufficient to say 'Use RAID' because within the Venn Diagram of things falling under RAID is a whole bunch of shit solutions and awful experiences.
I haven't seen a machine shipped with firmware RAID in decades.
It's still enabled in the firmware of some vendors' laptops -- ones deep in Microsoft's pockets, like Dell, who personally I would not touch unless the kit were free, but gullible IT managers buy the things.
My personal suspicion is that it's an anti-Linux measure. It's hard to convert such a machine to AHCI mode without reformatting unless you have more clue than the sort of person who buys Dell kit.
In real life it's easy: set Windows to start in Safe Mode, reboot, go into the firmware, change RAID mode to AHCI, reboot, exit Safe Mode.
Result, Windows detects a new disk controller and boots normally, and now, all you need to do is disable Bitlocker and you can dual-boot happily.
However that's more depth of knowledge than I've met in a Windows techie in a decade, too.
After over a decade of in-kernel development, Btrfs still can't either give an accurate answer to `df -h`, or repair a damaged volume.
Because it can't tell a program how much space is free, it's trivially easy to fill a volume. In my personal experience, writing to a full volume corrupts it irretrievably 100% of the time, and then it cannot be repaired.
IMHO this is entirely unacceptable in an allegedly enterprise-ready filesystem.
The fact that its RAID is even more unstable merely seals the deal.
> Btrfs still can't either give an accurate answer to `df -h`, or repair a damaged volume.
> In my personal experience, writing to a full volume corrupts it irretrievably 100% of the time, and then it cannot be repaired.
While I get the frustration, I think you could have probably resolved both of them by reading the manual. Btrfs separates metadata & regular data, meaning if you create a lot of small files your filesystem may be 'full' while still having data available; `btrfs f df -h <path>` would give you the break down. Since everything is journaled & CoW it will disallow most actions to prevent actual damage. If you run into this you can recover by adding an additional disk for metadata (can just be a loopback image), rebalancing, and then taking steps to resolve the root cause, finally removing the additional disk.
May seem daunting but it's actually only about 6 commands.
Hi. My screen name is my real name, and my experience with Btrfs stems from the fact that I worked for SUSE for 4 years in the technical documentation department.
What that means is I wrote the manual.
Now, disclaimer, not that manual: I did not work on filesystems or Btrfs, not at all. (I worked on SUSE's now-axed-because-of-Rancher container distro CaaSP, and on SLE's support for persistent memory, and lots of other stuff that I've now forgotten because it was 4 whole years and it was very nearly 4 years ago.)
I am however one of the many people who have contributed to SUSE's excellent documentation, and while I didn't write the stuff about filesystems, it is an error to assume that I don't know anything about this. I really do. I had meetings with senior SUSE people where I attempted to discuss the critical weaknesses of Btrfs, and my points were pooh-poohed.
Some of them still stalk me on social media and regularly attack me, my skills, my knowledge, and my reputation. I block them where I can. Part of the price of being online and using one's real name. I get big famous people shouting that I am wrong sometimes. It happens. Rare indeed is the person who can refute me and falsify my claims. (Hell, rare enough is the person who knows the difference between "rebut" and "refute".)
So, no, while I accept that there may be workarounds that a smart human may be able to do, I strongly suspect that these things are accessible to software, to tools such as Zypper and Snapper.
In my repeated direct personal experience, using openSUSE Leap and openSUSE Tumbleweed, routine software upgrades can fill up the root filesystem. I presume this is because the packaging tools can't get accurate values for free space, probably because Btrfs can't accurately account for space used or about to be used by snapshots, and a corrupt Btrfs root filesystem can't be turned back into a valid consistent one using the automated tools provided.
Which is why both SUSE's and Btrfs's own docs say "do not use the repair tools unless you are instructed to by an expert."
Hey. That's sounds like an awful experience. Btrfs has some rough edges especially since a lot of maintenance tasks are "manual", and you are right to try and address it. And it's annoying that becomes personal for some people with too much time.
For my perspective, my experience with btrfs has been flawless through 11 machines and at least 3 major releases on each without any maintenance but it could just be I'm not hitting the worst case (only use snapshots on 2 machines, raid on 3). And I've only used btrfs since fairly recently (~4 years now). I've had to recover one drive of a friend using the method I outlined before as he filled the entire drive with media. For me the trade-off of a few rough edges but more functionality & flexibility than other filesystems is worth it.
For your update issue, I think you're mostly correct; the package manager likely assumes the filesystem is not snapshotted (i.e. it will reclaim disk space), while btrfs with snapshots/CoW will use the entire size of written files unless it's in the same snapshot.
I worked at Red Hat very briefly, and for SUSE for longer than ever before. Both were good workplaces with a good atmosphere: RH is one of the friendliest places ever, and I'm still friends with former colleagues from over a decade ago.
OTOH, installing Fedora 14 was like having a bucket of cold water to the face. I used and reviewed Red Hat Linux in the 1990s and it was a massive PITA. It had no automatic dependency resolution, so complex software installation (e.g. going from KDE 1.x to KDE 2.x) was a huge task involving manually installing hundreds of dependencies.
RH would not bundle KDE (because Qt was not 100% FOSS) -- which is also why Mandrake was founded -- and so the GUIs on RHL were poor.
I switched to SUSE. Good package management, good GUIs, good system-management tools. (YaST was way better than RH's inadequate `linuxconf`.)
RH "fixed" this by... removing Linuxconf.
Trying Fedora a decade later and it was just as bad. All the external bits improved because upstream improved. GNOME was still a mess. KDE had got much more bloated. Xfce was better than ever.
But the RH in-house bits, while having nice visual design, were functionally terrible. The installer was an embarassment.
Over a decade of work since I reviewed RHL 9 and it was worse than ever.
A few years later, go to work at SUSE, and hey, openSUSE was lovely. All the good bits still there and improved. Yeah, a bit bigger and slower and clunkier than Ubuntu.
But there's always a downside.
An older team so less party atmosphere. Fewer "team building" sessions in the pub.
And while I was away, SUSE switched from ReiserFS to Btrfs, and as usual, SUSE of old being fond of experimental bleeding-edge filesystems, it's half-implemented and doesn't work right.
Snapper doesn't prune snapshots thoroughly enough. It fills your disk and because of unimplemented or non-working features it can't tell when this will happen.
Official SUSE answer: give it lots of space. Here, our FS falls over unrepairably when full, so give it all your space so it won't fill up! And you can't repair it so take lots of backups!
Every distro has downsides. Every filesystem has downsides but they are much less obvious.
Btrfs made my openSUSE boxes collapse and crash 2-3 times a year for 4 years. That is intolerable. I put up with that kind of crashy junk in 1997 or so but not 20 years later.
This is incorrect, quoting Linux 6.7 release (Jan 2024):
"This release introduces the [Btrfs] RAID stripe tree, a new tree for logical file extent mapping where the physical mapping may not match on multiple devices. This is now used in zoned mode to implement RAID0/RAID1* profiles, but can be used in non-zoned mode as well. The support for RAID56 is in development and will eventually fix the problems with the current implementation."
I've not kept with more recent releases but there has been progress on the issue
I believe RAID5/6 is still experimental (although I believe the main issues were worked out in early 2024), I've seen reports of large arrays being stable since then. It's still recommended to run metadata in raid1/raid1c3.
We bought a few Kioxia 30.72 TiB SSDs for a couple of thousand in a liquidation sale. Sadly, I don't work there any more or I could have looked it up. U.2 drives if I recall, so you do need either a PCIe card or the appropriate stuff on your motherboard but pretty damn nice drives.
Not really. I know that my sleep is worth more than the difference between HDD and SSD prices, and I know the difference between the failure rates and the headache caused by the RMA process, so I buy SSDs.
In essence, what we together are saying is that people with super-sensitive sleep that are also easily upset, and that don't have ultra-high salaries, cannot really afford 18 TB of data (even though they can afford an HDD), and that's true.
Well, again, well done on being able to afford it. I have 24TB array on cheap second hand drives from CEX for about £100 each, using DrivePool - and guess what, if one of them dies I'll just buy another £100 second hand drive. But also guess what - in the 6 years I had this setup, all of these are still in good condition. Paying for SSDs upfront would have been a gigantic financial mistake(imho).
Might be a bit adventurous for primary storage (though with enough backup and redundancy, why not). But seems perfect for me for backup / cold storage.
There's a big difference between used as in I just bought this hard drive and have used it for a week in my home server, and used as in refurbished drive after years of hard labor in someone else's server farm
Enterprise drives are way different than anything consumer based. I wouldn't trust a consumer drive used for 2 years, but a true enteprise drive has like millions of hours left of it's life.
Quote from Toshiba's paper on this. [1]
Hard disk drives for enterprise server and storage usage (Enterprise Performance and Enterprise Capacity Drives) have MTTF of up
to 2 million hours, at 5 years warranty, 24/7 operation. Operational temperature range is limited, as the temperature in datacenters
is carefully controlled. These drives are rated for a workload of 550TB/year, which translates into a continuous data transfer rate of
17.5 Mbyte/s[3]. In contrast, desktop HDDs are designed for lower workloads and are not rated or qualified for 24/7 continuous
operation.
From Synology
With support for 550 TB/year workloads1 and rated for a 2.5 million hours mean time to failure (MTTF), HAS5300 SAS drives are built to deliver consistent and class-leading performance in the most intense environments. Persistent write cache technology further helps ensure data integrity for your mission-critical applications.
Take a look at backblaze data stats. Consumer drives are just as durable, if not more so than enterprise drives. The biggest thing you're getting with enterprise drives is a longer warranty.
If you're buying them from the second hand market, you don't likely get the warranty (and is likely why they're on the second hand market)
There isn’t a significant difference between “enterprise” and “consumer” in terms of fundamental characteristics. They have different firmware and warranties, usually disks are tested more methodically.
Max operating range is ~60C for spinning disks and ~70C for SSD. Optimal is <40-45C. The larger agents facilties afaik tend to run as hot as they can.
> drive has like millions of hours left of it's life.
It doesn't apply for the single drive, only for a large number of drives. E.g. if you have 100000 drives (2.4 million hours MTTF) in a server building with the required environmental conditions and maximum workload, be prepared to replace a drive once a day in average.
Drive failure rate versus age is a U-shaped curve. I wouldn't distrust a used drive with healthy performance and SMART parameters.
And you should use some form of redundancy/backups anyway. It's also a good idea to not use all disks from the same batch to avoid correlated failures.
