“There are two types of people: those who back up and those who haven’t lost data … yet.”

• a few git repos (pushed and backup in the important stuff) with all docker compose, keys and such (the 5%)

Um, maybe I'm misunderstanding, but you're storing keys in git repositories which are where...?

And remember, if you haven't tested your backups then you don't have backups!

In 2010 I self hosted a Xen hypervisor and used it for everything I could. It was fun!

I had a drive failure in my main raid 5 array so bought a new disk. When it came to swap it out, I pulled the wrong disk.

It was hardware raid using an Adaptec card and the card threw the raid out and refused to bring it together again. I couldn't afford recovery. I remember I just sat there, dumb founded, in disbelief. I went through all the stages of grief.

I was in the middle of a migration from OVH to Hetzner and it occurred at a time where I had yet to reconfigure remote backups.

I lost all my photos of our first child. Luckily some of them were digitised from developed physical media which we still had. But a lot was lost.

This was my lesson.

I now have veeam, proxmox backup server, backuppc and Borg. Backups are hosted in 2 online locations and a separate physical server elsewhere in the house.

Backups are super, SUPER important. I've lost count how many times I've logged into backuppc to restore a file or folder because I did a silly. And it's always reassuring how easy it is to restore.

I have all my photos on my NAS whith backups to OVH storage for recent data and AWS glacier for folders of pictures older than two years. I can afford to lose all my ISOs but not all the pictures of my child. Never tried to recover data from AWS glacier but I trust them.

Plus, even if you manage to never, ever have a drive fail, accidentally delete something that you wanted to keep, inadvertently screw up a filesystem, crash into a corruption bug, have malware destroy stuff, make an error in writing it a script causing it to wipe data, just realize that an old version of something you overwrote was still something you wanted, or run into any of the other ways in which you could lose data...

You gain the peace of mind of knowing that your data isn't a single point of failure away from being gone. I remember some pucker-inducing moments before I ran backups. Even aside from not losing data on a number of occasions, I could sleep a lot more comfortably on the times that weren't those occasions.

Tools like restic and Borg and so critical that you will regret not having had them sooner.

100000%

I just experienced this when a little mini PC I bought <2y ago cooked its nvme and died this month. Guess who has been meaning to set up backups on that guy for months?

Unfortunately, that nvme is D. E. D. And even more unfortunately, that had a few critical systems for the local network (like my network controller/DHCP). Thankfully it was mostly docker containers so the services came up pretty easy, but I lost my DBs so configs and data need to be replicated :(

The first task on the new box was figuring out and automating borg so this doesn't happen again. I also set up backups via my new proxmox server, so my VMs won't have that problem too.

Now to do the whole 'actually testing the backups' thing.

I think these kind of situations are where ZFS snapshots shine: you're back in a matter of seconds with no data loss (assuming you have a recent snapshot before the mistake).

Edit: yeah no, if you operate at the disk level directly, no local ZFS snapshot could save you...