As most people know, I’m a digital hoarder. I never delete anything. I have around 4.6 terabytes of data stored in my Google Drive alone. That’s cool and all but it becomes interesting when I start looking at backup solutions for my data.
One of the best solutions out there (in my opinion) for data archiving and data backup is yet another product from Amazon Web Services called Amazon Glacier.
Glacier let’s you store your data securely for a whopping $0.004/gb/month, however there are drawbacks. Since this service is meant for data archiving, it is stored in what’s known as “cold storage” meaning your data is not accessible on-demand. Instead, you (or more likely, your application) will tell Glacier that you would like to download a certain file from your “vault” (your archive) and then 3-5 hours later (unless you pay for expedited), your application will receive a notification that your file is ready for it to download and it has 24 hours to do so.
Another catch is that even though it will let you download the entirety of your vault as fast as you can download it, it will cost you. To get your data back out of Glacier, it costs an additional $0.0025-$0.03/gb. That may not sound like a lot but when we get to talking about terabytes or petabytes of data, it adds up quick.
To sum up, I still think that Amazon Glacier is a great product if used correctly. For instance, if by law your organization is mandated to keep archives for x number of years and you know the chances of actually having to dig them up one day is slim? Glacier is perfect. Or as a last resort backup, meaning you have two or three other backups you can try to extract your data from before you have to dig into Glacier, then yeah.
Most of the Linux servers that I setup and/or maintain I typically run in CLI (command line interface) mode and don’t even bother to install a GUI just to preserve disk space, memory usage and CPU usage. However, it can be a pain to access my cloud storage services.
Then a few years ago, I discovered RClone. RClone lets you perform basic file and directory functions such as copy, move, delete and sync. I primarily use my Google Drive but RClone supports:
- Google Drive
- OpenStack Swift
- Amazon S3
- Google Cloud Storage
- Amazon Drive
- Backblaze B2
- Yandex Disk
One of the reasons why I love RClone is how simple it is to use:
- Download it from http://rclone.org/downloads
- Extract it using
tar -xvf rclone*
- Run it for the first time using
- It will then prompt you for the connection type, I choose 8 for Google Drive
- Then it will prompt you to name the connection. I keep it simple by just naming mine “g”
- It will then ask you to authenticate
- Confirm your settings and you’re done
Now to use it, for example to copy a file to your cloud storage, just use:
/path/to/rclone/rclone copy localfile (name of your storage):/
In my case:
/home/pi/rclone/rclone copy myfile.ext g:/
It still amazes me how many people don’t know about Google Photos. Google Photos is a free app and service for iOS and Android that backs up all the pictures from your phone to your existing Google account—and its unlimited! I highly recommend it to everyone just to have a backup for all of their pictures. Now the free version is unlimited but it does slightly reduce the quality of the photos but it’s so minimal that most people won’t see a difference.
Aside from just having a backup, Google Photos is a great way to free up space on your phone. Once you have all of your pictures uploaded, you can confidently delete them from your phone having peace of mind that they’re backed up on Google’s servers.
Amazon Prime Photos
Another service that people overlook is Amazon Prime Photos. Most people nowadays are Amazon Prime members, but one of the benefits of being a member is they offer unlimited backup of photos from your iOS and Android device. And unlike Google Photos, they backup your photos at their original quality.
If you have a Dropbox account, you can use the service to automatically backup your photos on your iOS or Android device for free. Granted, a Dropbox free account only provides you with two gigs of free storage, but what some people do is use it until it’s completely filled up and then moving all of the pictures to a flash drive or an external hard drive for safe keeping.
Those are just three ways to freely have continuous backups of all of your photos. There is no reason why you can’t use all three of these services at the same time, in fact, I recommend it! The typical rule of thumb is your files aren’t completely backed up until there are three different copies in three different locations.
It still amazes me that this day in age, that you need to remind people to backup their computers, or more commonly, try to convince people that backing up your important files once every six months to a flash drive or external hard drive isn’t exactly the best strategy.
Now, you definitely have no excuse. Earlier this month Google released their new desktop client for Google Drive. The most important feature that it added that the client lacked before was the ability to automatically and continuously backup files on your desktop, documents folder, etc. Just download once, set it and forget it, the client does the rest. Best of all, every Gmail user already has 15 gb free! It’s not much but it’s plenty enough to backup impo documents and such.
Even if you already have backups, why not just add one more layer on top of it for extra protection? Go download the Google Drive client for Mac or PC and get it setup and enjoy peace of mind.
Sometime last year right around the time that Dropbox had their database of user accounts and passwords compromised, a client of mine got wind of the story and asked me to remove all of their cloud services to on-premise servers.
At the time, I was using Dropbox Plus to keep their network drive in sync between their multiple business locations (quite honestly because it just worked and it was one last thing that I had to worry about managing) so I began researching open source, self hosted alternatives to Dropbox. I quickly came across OwnCloud which is exactly what I was looking for. OwnCloud runs on top of your existing LAMP stack and has a web client, desktop sync clients as well as mobile clients for both iOS and Android. Best of all, it has file versioning built in. You can also use the EFF’s LetsEncrypt to secure the data in transit using SSL.
Ever since discovering it, I’ve also ran my own personal OwnCloud server using a Raspberry Pi and a one terabyte external hard drive. Of course, with any self hosted service, you get the responsibility of backing it up. I don’t keep anything mission critical on my OwnCloud but I wrote a simple Python script to copy over all of the data, dump the MariaDB database, tar it into an archive, send it through an encryption process and send it up to my Google Drive once a week.
Even if you don’t have a business, having your own personal cloud storage is still a fun project to do. For less than $99 (the cost for 1 year of Dropbox Pro), you can go on Amazon and order yourself a cheap Raspberry Pi kit as well as a 1 terabyte USB external hard drive and build your own personal cloud storage!
So I’ve used Amazon Cloud Drive as my primary online service provider for the last few years. How could I not? I mean, it was $60/year for unlimited storage. Over the past few years I’ve managed to store 4.6 terabytes of data on their cloud drive. Backups, movies, pictures, music, you name it and I probably had it stored in there.
So you can imagine my disappointment when I logged into my account a couple of months ago and it said that my plan was changing from unlimited storage for $60/year to 5 terabytes for $299/year. And worse, I only had a month before my unlimited plan ended so I only had a month to find my data a new home. I lost faith in on premise external drives back in college when I dropped the external hard drive that contained my 200 gb of music and lost it all.
I took about a week researching and researching different online storage providers. The biggest problem I had was all of the reputable companies that I knew that would be around years from now, all maxed out at 1 terabyte. I finally gave up and decided just to upgrade my G-Suite Basic account ($5/user/month) to G-Suite Business ($10/user/month) so I could utilize the one terabyte storage in Google Drive and just move over the priority files.
How G-Suite Business accounts are supposed to work is that for accounts that have less than 5 users, each user is supposed to be limited to 1 terabyte per user and for accounts that have more than 5 users, each user gets unlimited storage.
I only have one user under my account (myself) so I was baffled when I went under my storage settings and it said that I had unlimited storage. So I decided to test it. I launched a Google Compute instance and started to transfer all of my data from Amazon Cloud Drive to Google Drive using Syncovory (so I could transfer 10 different files concurrently versus one at a time) and was able to transfer it all within a couple days.
Two months later, still no issues. So if you are looking to store large amounts of data, give G-Suite Business a look.