How to Create a Very Inexpensive Serverless Database
Cloud object storage can used as a powerful, very inexpensive database
Would you believe you can use a serverless database that is fully managed, massively scalable, highly available, and so inexpensive that for $5/month you can store hundreds of millions of records and read and write billions of records? You can if your database needs can be met with a very simple key-value store. The solution is to use inexpensive cloud object storage.
Amazon released their S3 (Simple Storage Service) object storage product in 2006. Since then every other major cloud service provider, and many other cloud providers, have offered an object storage solution.
Object storage can be used to store photos, videos, backups, etc., but can also be used to store data. Data can be stored in any format you want, such as JSON, BSON, XML, CSV, protocol buffers, MesssagePack, plain text, or any other data serialization format.
Objects are stored in buckets. Buckets can usually be created in data centers in specific regions around the world. This allows you to bring your data closer to the users that will be accessing the data. With some providers, buckets can be configured as websites, allowing you to use your object storage as an inexpensive website host.
Pricing
The main advantage of object storage over serverless databases is price. Storage generally costs 1¢–2¢ per GB and outbound network costs for the least expensive options are generally 1¢ per GB.
Many object storage products also charge for read and write API requests. If you are using object storage as a database solution, this may greatly increase your costs if you get hundreds of millions or billions of requests per month, so it’s best to avoid products that charge for read and write requests. There are plenty of solutions that do not charge for requests. In the table below, the Read and Write columns indicate cents per million requests. Products with a ‘-’ have no request fees.
For comparison, here are the same pricing values for some serverless databases. (See “Serverless Databases for Your Mobile or Web App.”) The budget object storage products are a fraction of the price of serverless databases.
The following table shows the monthly cost for a few object storage services and databases with various amounts of storage and read and write requests. The record size is assumed to be 500 except for the last two columns, which have a small 100 and a large 4000 record size. This shows that until you start storing and reading and writing hundreds of millions of records, the pricing actually varies little. Beyond that point, however, the product you choose can make a big difference.
Accessing Data
Most object storage products are compatible with Amazon’s S3 REST API. The two major exceptions are Google and Azure. S3 compatibility means you can use tools and SDKs that are built for S3. You can also easily switch providers, avoiding vendor lock-in, or use multiple providers just by changing the access URL.
The simple S3 REST API makes it ideal for use by mobile and web apps that can directly access the data without requiring a separate API server.
Objects are stored in buckets in a flat structure without the hierarchy you get with file systems. However, you can simulate this by using a path separator, which is normally ‘/’. For example, ‘photos/photo1.jpg’ creates a simulated ‘photos’ folder with ‘photo1.jpg’ inside that folder.
The S3 API allows you to read, write, and delete objects. Objects are accessed by their unique key, which may be similar to a file name. There is no capability of doing queries such as “SELECT * FROM Employees WHERE City=’San Diego’”. This may greatly limit how you can use object storage as a database, but there are plenty of applications where all you need is access to data by id.
With object storage it’s also not possible to do record updates. Any changes to a record are done by rewriting the entire record.
Buckets and objects have access privileges to limit who can read and write. Public access can be granted to buckets and/or objects to allow anyone to read or write.
Performance
Object storage performance is generally very good, even when compared with a regular database. I verified response times for a few object storage products and found times of 50 ms — 300 ms. I found similar results with serverless databases. Although databases can often access data in less than 10 ms, by the time the request and response travel the internet, the access time is greatly increased.
Rate Limits
Be aware that providers generally have a limit for the number of requests that can be made per second. The limits may be high enough that you won’t be affected. Not all providers specify their rate limits. Google is the only one that claims they will scale to meet your request rate.
AWS can read 5,500 and write 3,500 objects per second per prefix (folder). If you require more requests than that, split your data into folders.
Backups
Database backups are generally performed to avoid data loss in case of server failure or to restore data from user or application errors.
Database durability is a measure of how much the provider can guarantee that your data will not be lost and is usually expressed as a percentage with lots of 9’s. For example, 10 nines of durability would be 99.999999%. Of the major providers, AWS and Google claim 11 nines, and Azure claims 12. Durability is strengthen by making replicas of the data in data centers located in different geographic areas. Because of high durability, it may not be necessary to make backups of your object storage data.
Almost every object storage provider allows you to version your objects. This means that every time an object is replaced with an update, the older object is preserved with a version number, making it possible to go back to a previous version. This provides an automatic backup in case of user or application errors.
Product Details
AWS S3
AWS S3 is the gold standard by which all other object storage products are compared. Pricing of S3 is similar to other products in the same category from other providers.
- Pros: Part of the huge AWS system; stable and secure; very high rate limits; many data centers around the world
- Cons: Fees for requests
AWS Lightsail
Lightsail is a division of AWS that has servers and a few other cloud products that are priced to compete with budget cloud providers such as DigitalOcean. Lightstail products are also generally much easier to set up and use compared to regular AWS products.
In July 2021, Lightsail announced an inexpensive object storage product built on AWS S3. This is somewhat of a game changer. You seem to get all of the benefits of S3 at a much reduced price. Bundles are $1, $3, or $5 per month with storage of 5, 100, or 250 GB with 25, 250, or 500 GB of network transfer. Storage and network beyond those limits are charge at the normal S3 rate. However, there are no request fees either within or beyond the bundle limits.
The $5 plan puts S3 at a price similar to the budget plans offered by DigitalOcean, Linode, and Vultr. However, unlike the plans of the other budget providers, Lightsail bundles seem to be limited to a single bucket. If you need to set up multiple buckets in regions throughout the world, a good pricing strategy might be to set up several $1 buckets, then increase to $3 or $5 buckets as needed. Lightsail allows you to switching pricing plans once during the billing period.
- Pros: Same as S3 but at a much cheaper price; no request fees; low $1 minimum price
Azure Blob Storage
Azure object storage is called blob storage.
- Pros: Part of the huge Azure system; stable and secure; very high rate limits; many data centers around the world
- Cons: Fees for requests; not S3 compatible
Google Cloud Storage
Google Cloud Storage is Google’s object storage product.
- Pros: Part of the huge Google Cloud system, stable and secure, unlimited rate limits, many data centers around the world, always free options in 3 regions (5GB storage, 5,000 writes/month, 50,000 reads/month, 1GB network/month)
- Cons: Fees for requests after always free limits; highest network rates of any provider; not s3 compatible
Firebase
Firebase is a Google company that provides a lot of cloud products at reasonable prices that are generally easy to set up and use. With the generous free options, many companies can use Firebase Cloud Storage for free.
- Pros: Same as Google but with much better free options (20K writes/day, 50K reads/day, 1GB/day network)
Oracle Cloud
Oracle Cloud Object Storage has better pricing than the other large providers.
- Pros: Good value for write requests; no network fees for first 10TB/month and very cheap after
- Cons: Fees for requests; high storage pricing
Alibaba
Alibaba’s Object Storage Service (OSS) is a very cost-effective option. There is a pay-as-you-go plan or a plan with storage and network limits that may save a little on costs. There are 21 data centers with 10 of them in China.
- Pros: many data centers around the world; good pricing with no request fees unless you reach 500 million request/month
DigitalOcean
DigitalOcean is a smaller cloud company that provides servers and databases that are easy to deploy quickly and are priced below that of the large cloud providers. Their object storage is called Spaces and was the first to offer 250GB of storage, 1TB of outbound transfer, and no request fees for an amazingly low price of $5/month. Any storage or network above the limits is charged at a very lost price. This fee structure is ideal for using object storage as a very inexpensive database.
One issue with Spaces is rate limits. Each Space (bucket) is limited to about 200 requests/second. If that is a problem, you might be able to work around that by using multiple Spaces. You can create up to 100 Spaces within the same account. (I’ve found that some users have had problems accessing Spaces at rates far below 200/second.)
- Pros: Incredible value
- Cons: Rate limits may restrict some users; only 5 data centers
Linode
Linode is similar to DigitalOcean providing cloud services at great prices that are easy to deploy. Their object storage product is priced the same as DigitalOcean.
- Pros: Incredible value
- Cons: Only 4 data centers
Vultr
Vultr is similar to DigitalOcean and Linode. Their object storage product is priced the same as the other two.
- Pros: Incredible value
- Cons: Only 1 data center in NJ
Exoscale
Exoscale’s object storage is competitively priced with 6 data centers in Europe.
- Pros: Good value
- Cons: Data centers only in Europe
Scaleway
Scaleway’s object storage is very competitively priced with 3 data centers in Europe. You get 75GB of storage and network free very month.
- Pros: Extremely good pricing and free tier
- Cons: Data centers only in Europe
Wasabi
Wasabi does only one thing and that is object storage. They offer the best pricing of any other provider. There are no fees for reads, writes, or network. The only cost is for storage at $0.006/GB, which is extremely inexpensive.
However, the pricing FAQ reveals some details that may make Wasabi unsuitable for use as a database. Objects have a minimum duration of 90 days. If an object is deleted before 90 days, it is still counted as storage for the full 90 days. There doesn’t seem to be any documentation on what happens if objects are updated several times during a 90-day period.
Then there is this: “If your monthly egress data transfer is greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress policy.” And: “If your use case exceeds the guidelines of our free API request policy on a regular basis, we reserve the right to limit or suspend your service.”
- Pros: Overall best pricing
- Cons: Restrictions may prohibit using Wasabi as a database
Backblaze B2
Backblaze does computer backups and B2 object storage. B2 has the lowest price for storage at $0.005GB/month and very low download pricing at $0.01GB/month. It does, however, charge the typical $0.40 per million for read requests. If you’re not doing hundreds of millions or billions of requests per month, this may be a good option.
- Pros: Excellent pricing
- Cons: Read request fee; only 3 data centers; can’t select data center
Tebi
Tebi is a pretty new organization that does only object storage. Their main feature is live replication to other servers to bring the data closer to the users. Unfortunately, they currently only have 4 data centers.
- Pros: Great pricing; free tier of 25GB storage and 250GB network
- Cons: Only 4 data centers; pretty new so hasn’t been time-tested
Comments
I would love to hear your comments if you have used object storage as a database solution. I currently have only done this with one web app using Cloud Storage for Firebase and it works extremely well. I am currently evaluating object storage options for a product that will need quite a bit of storage with potentially hundreds of millions of requests per month.
