• Please review our updated Terms and Rules here

What is the best storage media?

6885P5H

Experienced Member
Joined
Feb 7, 2015
Messages
320
Location
Québec, Canada
Hello. I am assuming many of you guys keep backups of large amounts of data. What would you say is the best media to store data? Right now I have DVDs and an hard drive, neither of which I trust. DVDs have their problems and hard drives are known for failing at any moment. Is Blu-ray any good? It can store large amounts of data and apparently they cannot be scratched, which I doubt is true but anyway. Even if they are hardier than DVDs surely they can decompose like their lesser brethren right? You know, when oxygen enters the disc and rots the reflective material...
 
Best? Hollerith cards, stored properly. That's my opinion. I've given up on optical discs. Currently I'm using 8mm video tapes. I'm reusing tapes that were written twenty years ago which are not all readable anymore. So I expect I'll get less than 20 years, which is about what I got with optical disks. The best thing to do though is probably just to use hard disks or SD cards and make routine backups, relocating the backup drives away from the source data.
 
Depends. After you look at what you have and decide what you never really use, has no real long term meaning, or can be downloaded again as needed and get a total data size you can go from there. Long term I would avoid CDR/DVDR/BR disks since they are easy to scratch, crack, bitrot (unless you have some pretty good storage methods). Anything you pick will need to be moved to newer media every decade or so.
 
FWIW, I never had a problem storing CDs/DVDs while exercising a little care. I archived my disks in a cool, dry place with jackets on and never stack anything on them. Some I've had for over 30 years and they seem to be as good as the day I put them away. Bottom line is I'm okay with them.
 
Final word - magnetic media with redundancy. Optical backups are the devils playground. Buy a big hard drive, fill it up. Buy another one, dump the contents of the first drive onto it. Do it a third time if you're paranoid. Cheaper then optical backups, MUCH more reliable. And FAR less labor intensive, not to mention time consuming. Do rudimentary disk maintenance 3 - 4x a year, scan disk, etc. If one fails, repllace it. There is no other way, never mind better way. This is the final word. A bit of wisdom that dawned on me thanks to Lutiana.

I had read many years ago that magnetic storage was more reliable then optical. I chose to believe otherwise. I have grave regrets. I took every one of my CDs and DVDs, dumped their contents onto a big hard drive, and I ain't looking back. I tossed the carcasses into the trash.
 
Last edited:
I'm a big fan of tape, and not too long ago tape was reasonable for doing backups. (I had Travan 4 at home.) It had a great shelf life and had reasonable capacity, but now with hard drives being so big that is not as true.

Lately I've been using external hard drives. Hard drives also have a reasonably long shelf life; if you run them once in a while they will do scans in the background to find and rewrite weak sectors. To protect myself against OS or other corruption I use MD5SUM or use Zip files. Those are more for detecting errors; neither can correct errors. But that is what the other copy is for.

Whatever you choose, plan on having at least two copies of it. I have two copies of everything I care about, and an off-site copy. And also plan on migrating to another set of devices after a few years. As good as hard drives are, they are mechanical and will eventually fail. Recopying (and verifying the data) periodically is kind of required.

FLASH based devices are interesting because they have no moving parts, but FLASH bit rots ... if you leave a thumb drive sitting for too long it will lose bits. Plug them in once in a while and test them, at least to force the bits to be refreshed.
 
Clay tablets have over 4000 years of proven reliability. Why not make the walls of your library into your library?

Hard drives upgraded every few years seems the only reasonably affordable method for storing large amounts.
 
1/2" open-reel tape has shown to be good for at least 50 years, but the problem is having the equipment to read it--and the knowledge of how to interpret the data. 50 years ago, ASCII was not common, but for use by teletype. Same for punched cards.

If you value it, regenerate it every 10 years or so. Shouldn't be a big problem--storage capacity of devices keeps increasing.
 
I use Generation 3 LTO tapes for maintaining backups. Each tape holds 400gb and we are now at a point where the tapes are cheap and the SCSI, FC and SAS versions of the drives are now under $100.
 
But the cost of the media, not taking in to account the cost of the drive, exceeds that of the equivalent hard drive, even an external. And I imagine it's way slower then a hard drive. No offense, but I can't see the point. Tape at one time made sense I guess. But at least it's magnetic and not the accursed optical diseased Frisbees.
 
Modern hard disks are extremely reliable, and if you can keep them spinning continuously with nice clean power they'll run for a very long time. The problem of course is keeping them spinning in the face of power outages, PSU failures, system upgrades, etc. In my experience few drives have failed while running, compared to the number that were running fine but then failed after a cold boot (power cycle). Based on this, I suspect your best bet for storage is to build yourself a NAS or a file server using as many drives as you need to hit the storage capacity you need, using RAID1 or RAID10 (avoid RAID4/5, with large drives those are no longer good options). After 3-5 years or so, set up another one (growing capacity as needed) but keep the previous one on-line and use it to back up your new one. 3-5 years later, repeat, setting up a new unit #3, making previous unit #2 your new backup device, and retiring #1 which has by now given you 6-10 years of faithful service. If you're using drives with 3-year warranties, aim for replacements at around the 3 year mark. If you use drives with 5-year warranties, you can wait longer. Try not to buy all your drives at the same time from the same place, spread out your purchases so you'll get drives from different manufacturing lots.

That said, I don't do this myself. My own practice is to run a single large file server for 8-10 years or so (using RAID1 or RAID10 of course), backing it up to tapes which I then store somewhere reasonably far away. I trust tape for backups, but the key here is that they are *backups* of data which is normally available on a file server. I do not use tapes for off-line archival storage, not because I don't trust the tapes themselves but rather because I don't trust the tape drives (and the old software) to continue working, which may put me into a position where I decide I want to read a tape one day but no longer have working hardware/software to read it with. Tape is very durable and easy to transport, hard disks are more fragile.

Regarding the cost of tape, I use LTO4 tapes and drives right now, which are very reasonably priced and hold a lot of data. I typically get between 800 and 1000 GB per tape, so a 'full' backup is 7 tapes (then I use LTO2 tapes for incremental backups). This is getting to be a lot (it was 4 tapes when I started with LTO4) so I'm thinking it's about time to upgrade to LTO6 (LTO8 is the current generation, so bargains on LTO6 stuff should be easy to find). Even so, I'm still spending more on tapes and tape drives than it would cost to buy external USB hard disks. The problem is, I don't trust USB hard disks to actually work when I need them. I have had too many that just fail to power on when I go to access them. My backups need to be stored on something *more* reliable than an internal hard disk because I'll only need to access them if my already very reliable RAID1/RAID10 disk volumes have failed. Since external USB drives have proven themselves to be *less* reliable than internal drives, they are logically a terrible choice for backups.

In a nutshell, I am using disks as my primary storage because they are incredibly economical. I'm also using disks as my primary on-site backups via RAID1/RAID10 (yes I understand these aren't 'technically' backups but relax because); for off-site backups I have another copy of my data on LTO4 tapes which are kept somewhere far enough away that a fire, hurricane, or other disaster is unlikely to destroy both the primary and backup copies of my data. I also have incremental backups on LTO2 tapes which I don't bother to store off-site; those would be used only if I had to recover from data loss due to a malware outbreak, hardware failure, or stupid mistakes.
 
Last edited:
But what would you use to store data for 40 or 50 years? Surely not the cloud (i.e., other people's computers) or a continuously running RAID server. How about 75 or 100 years?
 
For 40 or 50 years I actually would use a continuously-running server (but using a 'consumer' definition of "continuous"). It's the only way I would trust myself to keep something usable that long -- to be using it every day, fixing issues immediately as they arise. If I wanted to recover data from a 40-year-old tape I would look at the amount of work required to get a 40-year-old system running (including finding everything I'd need among all the piles of junk), then I would get discouraged, and I would decide that I don't actually need to read this tape today, I should do some cleaning first. Which never seems to happen. I have boxes full of 40-year-old floppy disks that I've been meaning to archive but I never get to it because setting up a system to do it is a lot of work, and I'm a busy person. Maybe when I retire I'll have time for all this.

It turns out the most unreliable component in the entire chain is me, so I've set up a system that will work reasonably well despite me.

To be fair I wouldn't actually keep any single server running for 40-50 years, I would expect to replace/upgrade the server periodically. I started with a NetWare 5 server decades ago but lost most of that data due to a disk failure of my NSS volume, unrecoverable because I wasn't verifying that my backups were readable (dust on some of my optical disk platters due to being kept in the drive with shutter open for long periods). Then I had a Windows 2000 server, whose data was migrated to Windows 2003, then migrated to OpenBSD on the same hardware, then migrated to OpenBSD on new hardware. Migrating data has turned out to be reasonably easy and a good opportunity to verify that my backups are actually usable. Along the way I've gone from using 650MB magneto-optical disks (325MB per side), to DDS-1 (2GB per tape), to DLT2000 (15GB per tape), to DLT7000 (35GB per tape), to LTO1/2 (100/200GB per tape), to LTO3/4 (400/800GB per tape).

Cloud storage isn't quite where it needs to be yet for me to be comfortable using it, but I wouldn't be surprised if it eventually becomes normal, expected and ubiquitous in enterprise settings. At that point I'd probably be comfortable using it myself. For now I'm happy using my own servers and doing my own backups.
 
On such long time periods it becomes a philosophical debate whether the data is still relevant.
 
But what would you use to store data for 40 or 50 years? Surely not the cloud (i.e., other people's computers) or a continuously running RAID server. How about 75 or 100 years?

For 40 or 50 years I actually would use a continuously-running server... I have boxes full of 40-year-old floppy disks that I've been meaning to archive but I never get to it because setting up a system to do it is a lot of work, and I'm a busy person. Maybe when I retire I'll have time for all this.

On such long time periods it becomes a philosophical debate whether the data is still relevant.
More important is whether any of you old geezers are still going to be relevant in 40 to 50 years.
 
Back
Top