1Gb of reliable storage .. is it possible?

Discussion in 'OT Technology' started by piratepenguin, Nov 17, 2006.

  1. piratepenguin

    piratepenguin New Member

    Joined:
    Jun 18, 2006
    Messages:
    1,067
    Likes Received:
    0
    Location:
    Ireland
    Ok, it's time I for once and for all sorted out a decent backup solution, so in 50 years I can look back at the shit I've done. Right now I'd have to go through tens, easily a hundred of CDs to get to my two year old crap, where I probably won't find what I was looking for.

    My important stuff in any era doesn't touch one gigabyte, so I want one gigabyte of storage. And it's gotta be seriously reliable - not in the nuclear-disaster sense, but in the data-corruption and that bullshit sense.

    I don't think I want or need anything fancy like harddrives.. More like flash cards or zip disks (what's reliable?).

    I thought about just buying (as in renting) somewhere online to upload my stuff to, but hmmmmm I dunno - anyone know/recommend any specialised services (that don't cost a tonne, or that do cost a tonne just for interests sake)?

    I could store it in my gmail (email it to myself, I often do that with some important stuff immediately after writing it, with subject "PIRATEPENGUIN, YOUR BACKUP IS READY" for searching purposes :)), there's even a filesystem driver (google for gmailfs) for Linux to do this pretty naturally (not sure if I'd trust it though). That's all clunky though, I'd prefer something cleaner.

    I had an equally messy idea to somehow utilize P2P to setup a network of people storing and sharing eachother's important files (oh yeah, extremely little of my (REAL) stuff needs to be kept outta the general population's hands), but things like updates would be equally tricky, and it'd probably need more luck than even my Irish ass can produce to pull off :big grin:

    Something I can mount and put together a very-snazzy, very-me-specific, backup script (of course everyone will be able to pick the script up at my site and tailor it for themselves (as long as they contribute back their changes, GNU GPL FTW!), and maybe one day it'll turn into a general thing that loadsa people use, but I'm not looking at that right now. I'll have to figure out what to do with MySQL, CVS/SVN and all that - that'll be the hardest thing about writing it) for, probably using rsync or something like that for doing the copying (incremental, etc), and have it run every few hours (hopefully it won't cause a ***MASSIVE*** performance hit, I don't think it will).

    So yeah, I wanna spend a lot of time at this.. since I want it to last me forever.

    What mediums should I look into? I don't wanna spend too much money - maybe a few hundred euro if I find something REALLY good. Scripts like what I've talked about can't possibly be a new idea .. any pointers to similar projects/"products" for that?
     
  2. Doc Brown

    Doc Brown Don't make me make you my hobby

    Joined:
    Mar 31, 2006
    Messages:
    16,404
    Likes Received:
    0
    Location:
    Ohio
    I think you're more concerned with the changing technologies than you need to be.

    Burn it to a dvd now.

    At whatever point dvd gets replaced by something else, you will still have a generation of motherboards that will allow for the dvd drive along with the next gen stuff. At that point you will just transfer everything over.

    Lather, rinse, repeat.


    Oh, and in the meantime you just keep the stuff in a folder on your hard drive.

    I use a couple of backup drives for my critical stuff, as well.
     
  3. EvilSS

    EvilSS New Member

    Joined:
    Jun 11, 2003
    Messages:
    5,104
    Likes Received:
    0
    Location:
    STL
    The problem isn't so much the media but the ability in 50 years to read it. There is archive quality media (DVD-RAM, for example, is designed to have a 30 year shelf life) but in 30 years, will you have a drive that can read it or a program that can handle for format it's in now? If you store it online how long do you think the company/network you store it on will be around? They come and go. I wouldn't count on even GMAIL being stable for 50 years.

    You pretty much have to plan on switching it out every 5 years or so. So get some archival DVD-R/+R's, or some DVD-RAM if your drive supports it and burn two copies (keep one at work or somewhere else where it will be safe if your home is vaporized in the next alien attack) and just remember to update it every two years or so, making complete new copies on whatever the popular, stable, storage medium is at that time.
     
  4. Doc Brown

    Doc Brown Don't make me make you my hobby

    Joined:
    Mar 31, 2006
    Messages:
    16,404
    Likes Received:
    0
    Location:
    Ohio
    Another thing that I think is likely is that drive storage is going to get more and more reliable as time goes on.

    At least I hope so!
     
  5. piratepenguin

    piratepenguin New Member

    Joined:
    Jun 18, 2006
    Messages:
    1,067
    Likes Received:
    0
    Location:
    Ireland
    Thanks for the replies, but I really hate using CDs and DVDs because I want something that'll keep itself up-to-date to the few-hours or the day (often, when I'm up to nothing useful, it'll have nothing to update, but it should always check and only ever update/replace changed files (that's why I think I might like rsync, but I've never used it myself before and CPU usage might be an issue)). That's no problem with a flash card (or similar.. I don't understand the differences between any those memory cards), but I can't simply mount and write to a CD or DVD. Rewriting the entire disk isn't fun, and neither is writing to a new disk.

    DVD-RAM is interesting, but it probably has these issues too. It'll be handy when it comes to archiving my stuff, for sure.

    Harddrives.. they FAIL! Double (at least) harddrives could work, but corruption, UGH. I gotta at least look for something more reliable.

    And yeah I realise I'll have to swap media (esp. the archives) at least a few times if this stuff will last 50 years, I'll do that happily.

    btw, if I go with some sorta memory card (512Mb would probably do it), I'd definitely keep two up to date.
    Using free software and open formats pretty much exclusively, I think I've a better chance than anyone with that ;)
     
  6. Doc Brown

    Doc Brown Don't make me make you my hobby

    Joined:
    Mar 31, 2006
    Messages:
    16,404
    Likes Received:
    0
    Location:
    Ohio
    Well the memory card is not a bad idea. They're cheap now, and if you back it up to two of them (different brands, just in case), you would probably be safe in the short term.
     
  7. 7960

    7960 New Member

    Joined:
    Oct 17, 2004
    Messages:
    60,415
    Likes Received:
    0
    Location:
    New England
    You want storage for 1G? Get a 2G smart media or CF card and adapter. As long as there's USB it'll work, and USB is going to be around for a long time.

    Software, try SyncBack

    http://www.2brightsparks.com/syncback/syncback-hub.html

    There's a free version. You can set it to work in 'off' times or when the PC isn't busy. You can have it only backup files that have changed......it'll do just about everything you mentioned wanting to do.
     
  8. EvilSS

    EvilSS New Member

    Joined:
    Jun 11, 2003
    Messages:
    5,104
    Likes Received:
    0
    Location:
    STL
    Flash memory is a bad idea. It is not considered archive media and they have a limited life span (they can only do so many read-write cycles before they begin to fail). They are great for temporary storage but I would not trust them to last.

    As for hard drives, again, you are stuck trying to actually read them later on. I mean, how likely is it that you could read a MFM or RLL drive on a modern computer today? And those were the standard 15 years ago.

    Your best bet is to use something transient (memory card, hard drive) and back it up to a more stable format every x days/weeks/months/whatever. Easily automated. You can even use rewritable media (DVD-RAM is great for this since it also has built in hardware level error correction) so you can update the files.

    As for open standards, sure, but don't go drinking the kool-aid on this. Over time even open formats will be abandoned for newer ones. I would not trust being able to read any format today in 50 years.
     
  9. Kieffer87

    Kieffer87 Orly OT Supporter

    Joined:
    Sep 23, 2005
    Messages:
    1,491
    Likes Received:
    0
    Location:
    Where the Green Grass Grows
    you could try GDrive by google that stores your files on your email server but puts a virtual drive in My computer for ease of use now. Then again who knows if they will change all that in several years or not?
     
  10. piratepenguin

    piratepenguin New Member

    Joined:
    Jun 18, 2006
    Messages:
    1,067
    Likes Received:
    0
    Location:
    Ireland
    XHTML and SVG I would hope so. ODF I would hope so.

    Maybe I'll just run Firefox 2 & OOo in an x86 emulator :big grin:

    Most stuff I work on these days are human readable anyhow so...

    I'm just not fond of anything I ever work on going into obliteration, but I'll especially need this stuff in the near future when my disks get unhealthy.

    So I'll work my way to creating or finding a suitable script. For now, since I have two harddrives, I can use it to keep my valuable stuff on both. When everything's working well I'll get two 1Gb memory cards (anything that works - I still don't know which is best in terms of reliability, but I should still be fine with two) - about €40 each (max), and keep my stuff updated on them.

    This is something I've wanted to sort out for a loooong time. If my primary disk failed right now the forums I (kinda) run would be gone back weeks (not a HUGE problem - not much happens there, but I'd rather we don't lose everybody's posts). Yeah, I should backup to CD every week or so, but that takes time. Backing up databases I haven't been able to automate, I'll try and figure it out now.

    That sure looks like a neat program 7960, but it doesn't run on my OS and (believe it or not) I need more features :big grin: neat program, but. It's like something every OS should include (I know the Ubuntu guys are kinda working on it, probably won't be as featureful as that, that's impressive.).

    GDrive is interesting too, but as before it won't run on GNU/Linux. And well I'd be a small bit weary - I'm not a HUGE fan of google anymore.
     
  11. EvilSS

    EvilSS New Member

    Joined:
    Jun 11, 2003
    Messages:
    5,104
    Likes Received:
    0
    Location:
    STL
    You would be surprised. There are lots of formats that were popular years ago that are unreadable now due to dropped support. Going that far out you have to consider that massive shifts in computing will have occurred. Just look at things like RIP Script. A popular, emerging graphics interface format for BBS's, then poof!, along comes HTML and the Internet. Now it's dead and you would have to dig through software archives to even find anything that can render it now.

    Now a practical suggestion: Why not mirror your hard drives? That way if one fails, the other will still have all the data intact, no lost time. Just replace the dead one and reinitialize the mirror. RAID cards and drives are cheap these days.
     
  12. piratepenguin

    piratepenguin New Member

    Joined:
    Jun 18, 2006
    Messages:
    1,067
    Likes Received:
    0
    Location:
    Ireland
    Any protection against losing some data to slow & silent corruption?

    I don't understand harddrives (and filesystems) and health at all.. In my experience, it's POOF and we won't let you read here or here anymore, and you only figure that out when it's too late for your more important stuff that you haven't touched in weeks.

    This may be solvable in RAID if it compares the mirror to the original every so often - can that be done?
     
  13. RyanL

    RyanL OT Supporter

    Joined:
    Nov 30, 2004
    Messages:
    4,584
    Likes Received:
    0
    Location:
    St. Paul, MN
    here is what i am doing on *nix

    i have an account with dreamhost that gives me 200gb of space.
    every night at 3am i have a script runs that uses rsync to sync to home folder to a folder on my dreamhost account. it works great and was insanely simple to set up.
     
  14. EvilSS

    EvilSS New Member

    Joined:
    Jun 11, 2003
    Messages:
    5,104
    Likes Received:
    0
    Location:
    STL
    Data usually doesn't get corrupted by the drive unless it's failing, then it usually just POOF, drive no longer workey. Modern drives vary rarely fail sector by sector. That used to be a problem like 10-15 years ago, but not so much today.

    Mirrors perform each data write twice, once on each drive. If one drive fails, the data still exists on the other one. Some systems allow you to turn on data verification, where the card writes the data then reads it back, verifying it's correct on the disk. Like I said though, it's unlikely that the physical disk will corrupt the data. That usually occurs due to a software issue, and the drives have no way to know that what is being written is wrong. That's why you back up to a second media periodically so you have historical copies that can't be changed.
     
  15. dorkultra

    dorkultra OT's resident crohns dude OT Supporter

    Joined:
    Oct 14, 2005
    Messages:
    22,735
    Likes Received:
    26
    Location:
    yinzer / nilbog, trollhio
    you need to store it on more than one form of media.

    my dad is a dentist, he uses a program called tdocs that holds all his patients info and billing stuff etc. it took about 1 year to input all the information and would take about 5 years to reproduce everything if lost.

    i have him copy over the entire program folder (it runs out of one folder, with the exception of a couple of system files - but they are not affected unless they are upgraded). he puts it on a flash drive and brings it home. i have the loaded onto my network server (just a linux box on my network, nothing special). i don't overwrite anything, space is cheap and the program is not large. the server has no intenet access, so it is hipaa compliant

    once a month i will burn a dvd with the entire months backups on it. this is placed in a fire resistant safe or his safety deposit box, alternated each month.

    i also keep backups on tape every 6 months and backups on hard disk every year. i'm very critical of keeping good backups, because it could ruin his practice if any info is lost.

    i can't make hard copies of the data, but my dad obviously has written dental records at his office.
    hard copies of important data is always a great idea
     
  16. deusexaethera

    deusexaethera OT Supporter

    Joined:
    Jan 27, 2005
    Messages:
    19,712
    Likes Received:
    0
    Store your important documents on paper.

    Store your important photos on negative film. There are companies that will burn your digital photos onto negatives for you.

    Store stuff that doesn't belong on paper on DSS tape or Magneto-Optical discs. MO discs work like CDRWs do, but they're massively-better-quality. They're used to store hospital and court records. They're not cheap, but trust me, recordable DVDs are NOT intended for long life -- they'd never make any money if they were.
     
  17. piratepenguin

    piratepenguin New Member

    Joined:
    Jun 18, 2006
    Messages:
    1,067
    Likes Received:
    0
    Location:
    Ireland
    I'll hopefully (at least) end up doing this with an rsync.net account. $1.80 for 1GB per month.. great.
    I don't have any important photos, but that's an interesting idea.
    Wow, I really got the idea that tape was obsolete, but a lot of people have been recommending it.
    What about DVD RAM? It seems to be the easiest to get, not that expensive either. (€9 for 3 4.7Gb disks and €45 drives should be ok, right?)

    I think I have my immediate backup medium solved now at least.

    I'll definitely go with RAID with my next machine.

    Thanks all :big grin:
     
    Last edited: Nov 20, 2006
  18. urmama

    urmama Uh Burrito? OT Supporter

    Joined:
    Oct 16, 2003
    Messages:
    6,273
    Likes Received:
    3
    Location:
    Texas
    :werd: and put them in a safe deposit box at your bank

    update when necessary :dunno:
     
  19. EvilSS

    EvilSS New Member

    Joined:
    Jun 11, 2003
    Messages:
    5,104
    Likes Received:
    0
    Location:
    STL
    If it's not already obvious, I'm a fan of DVD-RAM for archiving. It has a number of advantages of other media (which is why it's been replacing MO at a lot of companies as they cycle their hardware):

    - 30 year shelf life
    - Hardware data validation/error correction
    - Can be written to like a hard drive instead of being burned
    - Designed for 100,000 re-write cycles
    etc.

    Again though, just plan on upgrading media at some point. While the disc's may be readable in 30 years, there may not be anything readily available to read it with.
     
  20. deusexaethera

    deusexaethera OT Supporter

    Joined:
    Jan 27, 2005
    Messages:
    19,712
    Likes Received:
    0
    DVD-RAM does now what MO has been doing for 15 years. Just throwing that tidbit out there.

    Basically, MO has corporate sponsorship in the form of companies already owning the equipment -- I'm sure that plenty of businesses are replacing their equipment with the latest and greatest, but there are plenty more that will keep using MO for as long as suppliers will sell it to them. It's still going to be the more future-proof medium.
     
  21. EvilSS

    EvilSS New Member

    Joined:
    Jun 11, 2003
    Messages:
    5,104
    Likes Received:
    0
    Location:
    STL
    True, I don't expect MO to go away tomorrow. But it is dying off, mainly due to cost of media and drives when compared to DVD-RAM. My point is that DVD-RAM is just as reliable, and especially for an individual, much cheaper. It's like dot-matrix printers. They still exist but get more scarce and more expensive (really, price out a new one these days, yikes!) as they fade away. Between the two, I would not bet on MO being the "more future-proof". It is definitely in its declining years. Also keep in mind that DVD-RAM is not the "latest and greatest", well, not the latest at least. I remember the first unit I saw (at a health care provider no less) six years or so ago. Huge auto-changer. By huge I mean 7" cabinet that had 64 drives and storage for a large number of disks. It was new then, while MO was already approaching 15 years or so. Today MO is like Novell: There is still lots of it out there but no one is putting new installations in. There are just too many alternatives today. As companies replace hardware most are not buying MO anymore, they are either going with gargantuan tape libraries, SANs (big EMC stuff usually), or DVD-RAM. To be honest optical storage as an enterprise solution is going away period. We used to sell an early blue-laser optical storage unit as part of an email archiving solution. Today we still sell the archiving solution but most installations go to a dedicated SAN or offsite provider via tape.
     
  22. redrumkev

    redrumkev New Member

    Joined:
    Nov 10, 2003
    Messages:
    1,826
    Likes Received:
    0
    Location:
    Detroit area, MI
    What about an online solution that offers data backup? Also, this would be available through the internet, which I don't think is going anywhere in your lifetime!
     
  23. piratepenguin

    piratepenguin New Member

    Joined:
    Jun 18, 2006
    Messages:
    1,067
    Likes Received:
    0
    Location:
    Ireland
    Yeah, I'm gonna get an rsync.net account for that :big grin:

    And I'll also have archives on DVD RAM.

    Thank you all :)
     
  24. whatrick

    whatrick New Member

    Joined:
    May 5, 2006
    Messages:
    7,534
    Likes Received:
    0
    Location:
    locomotion
    i have gone through some of this as well recently.. at the moment I am still testing a backup solution that has so far been really good, looking to get a friend or family member to get it so we can swap backups as well thereby giving it catastrophic disaster redundancy, basically cheap offsite backup. -> http://www.vembu.com/

    the way I look at it.. when one drive fails, you will know it, and it gets replaced, and you will always have a backup. :dunno: .. unless they both die :noes:
     
  25. Vito_Corleone

    Vito_Corleone New Member

    Joined:
    Oct 12, 2003
    Messages:
    29,356
    Likes Received:
    0
    Location:
    Tampa, FL

Share This Page