Why I ditched MySQL and put Bob on DynamoDB

Over the past few years, I have all but given up on using MySQL whenever I need to write a database, just because I don’t like having to be careful about how many queries per second I can conduct without worrying about how much load the database server can handle at once and I have never liked the Oracle licensing arrangements.

Over the past few years, I have all but given up on using MySQL whenever I need to write a database, just because I don’t like having to be careful about how many queries per second I can conduct without worrying about how much load the database server can handle at once and I have never liked the Oracle licensing arrangements.

When I first started working on Bob years ago, I meant it to only be ran off of a single Raspberry Pi 3 which worked well for a while back when all Bob was doing was sending me a text message every eight hours and notifying everyone if I didn’t respond. During that time, the Raspberry Pi was serving as both the web server (Apache) as well as the database server (MySQL) which worked great at the time. However, as I started adding more and more functionality to Bob such as location tracking, social media checks, etc the MySQL service on the Raspberry Pi would crash, but even worse, it would silently crash so I could go a few days without noticing it was down. Not exactly what you want from a program that is supposed to be monitoring your life 24/7.

I eventually worked around the issue by lightening the load on how much data it stored and how often the scripts queried the data but it was a half ass fix.

So last month, when I decided to seriously work on Bob again, the very first decision I made was to ditch MySQL, and overhaul the backend to run exclusively on Amazon’s DynamoDB.

Why DynamoDB?

First of all, I’ve always been a huge fan of Amazon Web Services. Secondly, it’s a complete unmanaged solution. You create the tables and add the data and Amazon manages the rest.

When you create your tables, you specify how many reads and writes per second that each table needs to perform at and Amazon automatically spreads your data across how ever many servers that’s needed to support the specified throughput (we’ll come back to this).

By default, all tables only run off of solid state hard drives making it incredibly fast.

No Licensing Fees

Although it’s not open source, there are no licensing fees to use DynamoDB, you only pay for the hardware consumption that you provision per hour. For instance, if you know that your application will be heavily used during business hours during weekdays, you can provision to have more throughput during those hours and only get charged for those hours. Which brings me to my favorite feature of DynamoDB, auto scaling.

Auto Scaling

As I mentioned before, when you setup your tables, you get to specify how many reads and writes per second you want each table to handle but the truly beautiful part is its completely dynamic meaning you can adjust it throughout the day.

With old database models, you would typically have to think of your maximum expected capacity and run at that maximum capacity 24/7. With DynamoDB, you can specify a minimum and maximum read and write capacity and it will automatically scale up or scale back down based on usage.

For example, I have all of my tables set with a minimum read and write capacity 5 per second and a maximum of 10000 and have a rule where if at anytime, if 50% of my capacity is being used, double my capacity up until 10000.

What does this mean for Bob?

The more data we can collect, the more accurate algorithms can be.

Let me give you one example, on my personal account I have my computers reporting to Bob my usage based on mouse movement. When I had MySQL powering the backend, I had to build in a sleep mechanism where when it detected mouse movement, the computer would report it to Bob and then put itself to sleep for sixty seconds because otherwise, it would try to report to Bob multiple times per second and eventually overwhelm the database server. Now we can collect data up to milliseconds instead of minutes.

When you think of everything that’s either putting data into Bob, or taking data out: everything from computer check ins to motion sensor data to scripts that run every minute, on the minute 24/7, you start to see why MySQL started getting so overwhelmed.

So with the database bottleneck almost completely removed, I look forward to throwing as much data as possible into Bob!

Amazon Glacier: Great for Data Archiving or Last Resort Backups…But Nothing Else

As most people know, I’m a digital hoarder. I never delete anything. I have around 4.6 terabytes of data stored in my Google Drive alone. That’s cool and all but it becomes interesting when I start looking at backup solutions for my data.
One of the best solutions out there (in my opinion) for data archiving and data backup is yet another product from Amazon Web Services called Amazon Glacier.

Glacier let’s you store your data securely for a whopping $0.004/gb/month, however there are drawbacks. Since this service is meant for data archiving, it is stored in what’s known as “cold storage” meaning your data is not accessible on-demand. Instead, you (or more likely, your application) will tell Glacier that you would like to download a certain file from your “vault” (your archive) and then 3-5 hours later (unless you pay for expedited), your application will receive a notification that your file is ready for it to download and it has 24 hours to do so.

Another catch is that even though it will let you download the entirety of your vault as fast as you can download it, it will cost you. To get your data back out of Glacier, it costs an additional $0.0025-$0.03/gb. That may not sound like a lot but when we get to talking about terabytes or petabytes of data, it adds up quick.

To sum up, I still think that Amazon Glacier is a great product if used correctly. For instance, if by law your organization is mandated to keep archives for x number of years and you know the chances of actually having to dig them up one day is slim? Glacier is perfect. Or as a last resort backup, meaning you have two or three other backups you can try to extract your data from before you have to dig into Glacier, then yeah.

Freshbooks: Stop Creating Invoices in Microsoft Word

I’ve always been surprised of how many people have a business on the side either consulting or graphic design and the like and still use Microsoft Word to create and send their invoices. Stop it! There’s a better way! It’s called Freshbooks. I’ve used it for most of my business career and it’s quite honestly the best tool for the job.
With FreshBooks you can:

  • Create and send professional looking invoices
  • Have Freshbooks automatically generate and send recurring invoices
  • Track expenses and upload receipts
  • See when a client has or has not opened your invoice
  • Allow your clients to pay their invoices by credit card
  • Generate reports for profit and loss statements, expense reports, revenue by client, etc

I can’t say enough good things about Freshbooks, it is one of the best business tools that I’ve ever used! It starts at $15/month and you can try it free for 30 days. If you have to send out invoices every month, I highly recommend you check it out!

RClone: Tool for accessing your cloud via CLI

Most of the Linux servers that I setup and/or maintain I typically run in CLI (command line interface) mode and don’t even bother to install a GUI just to preserve disk space, memory usage and CPU usage. However, it can be a pain to access my cloud storage services.
Then a few years ago, I discovered RClone. RClone lets you perform basic file and directory functions such as copy, move, delete and sync. I primarily use my Google Drive but RClone supports:

  • Google Drive
  • Dropbox
  • OpenStack Swift
  • Amazon S3
  • Google Cloud Storage
  • Amazon Drive
  • OneDrive
  • Hubic
  • Backblaze B2
  • Yandex Disk
  • SFTP
  • FTP
  • HTTP

One of the reasons why I love RClone is how simple it is to use:

  • Download it from http://rclone.org/downloads
  • Extract it using tar -xvf rclone*
  • Run it for the first time using ./rclone config
  • It will then prompt you for the connection type, I choose 8 for Google Drive
  • Then it will prompt you to name the connection. I keep it simple by just naming mine “g”
  • It will then ask you to authenticate
  • Confirm your settings and you’re done

Now to use it, for example to copy a file to your cloud storage, just use:

/path/to/rclone/rclone copy localfile (name of your storage):/

In my case:

/home/pi/rclone/rclone copy myfile.ext g:/

Desktops-as-a-Service: Amazon Workspaces

One of the technologies that I’ve been fascinated with for the last few years is Desktop-as-a-Service or DaaS. This is where like most servers are moving to the cloud, individual workstations are also moving to the cloud slowly but surely.
One of my favorite services for this comes from infrastructure giant, Amazon Web Services with their Amazon Workspaces product. From $25/desktop/month (I’ll be the first to admit that it’s a bit pricey) you can have Amazon host your Windows 7 desktops.

There’s several reasons why I’m excited about this:

  • Zero reliance on individual hardware. Instead of buying each employee a new computer every few years to refresh their hardware or dealing with hardware breakdowns, all your desktops are safe in the cloud. For local clients you can either A. Recycle old computers and configure them as thin clients or B. Buy new thin clients for a couple hundred dollars for each workstation.
  • Minimal usage on your local internet connection. I can’t believe that I’m saying this in 2017 but there are still some businesses that can only get low bandwidth internet connections because of their location such as a single T1 line. But if your desktops are in the cloud, the only thing that your local connection will be used for is viewing the remote session. This means activities such as web browsing, downloading files, backing up to a remote service, etc. are all performed using the DaaS provider’s internet connection, not your local connection.
  • Mobile ready. It is incredibly simple for your users to access their desktops on their personal devices. Whether it’s an iPad or their home computer, they just download the client app, login and they’re at their workstation from wherever they are.

Those are just some of the few reasons that I’m completely intrigued about this new trend. My hope is when services like Amazon Workspaces get more and more popular, the price per desktop will fall. Again, $25/desktop/month adds up pretty quickly if you have more than a handful of users but I can see it becoming more of a no-brainer solution if the cost were to drop down to $5-$10/desktop/month.

Digital Ocean: Great VPS for Personal Projects

When I first started my business, the first thing that I had to host for a client was their email and website. I forget how I found them but for a couple years I rented a VPS from a company called Server Intellect which I later upgraded to a full on dedicated server. And then I came across Amazon Web Services and used their EC2 service to launch and run servers for my clients whenever I needed to. In fact, I was the consultant that helped manage one of my local community colleges, (Taft College) transition from hosting their website on-premise to Amazon Web Services.
I’ve always preferred AWS over Microsoft’s Azure or Google’s Compute Cloud quite honestly just because I am so familiar with it and already had my account setup as well as server images for different setups that I had.

However, probably around a year ago I switched all of my personal stuff such as scripts and my website over to Digital Ocean. Although I still advise businesses to use Amazon Web Services just because they have so many more advanced options and integrations with their other services, I recommend Digital Ocean for people like me who essentially just want to tinker around or just need to have a VPS for personal use in the cloud.

Digital Ocean refers to its server instances as “Droplets”. One of the things that I love about them is that all of their droplets comes with SSD hard disks (and you can really tell). Also, their entry level prices are unbeatable, starting at $5 for a droplet. But probably the thing that I love most about them is the simplicity of their console. They make it ridiculously simple to spin up a new VPS in a matter of minutes.

So the next time you need a VPS, give Digital Ocean a look!