Comparing the price-to-performance ratio of serverless services in the public cloud

Graduate thesis, Professional Specialist in Computer Engineering, Algebra University College, 2023. Supervisor: Vedran Dakić.

The thesis is in Croatian, titled “Usporedba odnosa cijene i performansi serverless usluga u oblaku”.

The question

Serverless is a way of billing for cloud resources: you pay for what your code uses while it runs, and the provider handles capacity planning, scaling, load balancing and monitoring. Every cloud provider sells several such services, and platforms like Railway and Supabase build on AWS, Azure or GCP and add features on top. I wanted to make the choice simpler by comparing price and performance among the popular services in each category.

Four categories, two benchmarked

I sorted the services by purpose into static websites, APIs, databases and object storage. I benchmarked the first two on these hosts:

• Static websites: Vercel, Netlify, Render, and two object stores with website hosting switched on, AWS S3 and Azure Blob Storage.
• APIs: Railway, Render, Fly.io, Azure Functions and AWS Lambda, plus Azure App Service as a conventional container host, to see whether serverless itself hurts responsiveness.

A database benchmark would depend too much on the query code and the ORM, and the object storage offerings were very similar, often identical, so for those two categories I only described the options (DynamoDB, Azure SQL Database, PlanetScale, S3, Bunny and others).

Method

I built the same static page in six front-end frameworks: React, Next.js, Vue.js, Svelte, Angular and jQuery. The API exists in five frameworks across four languages: Flask (Python), Express (JavaScript), Gin and Fiber (Go) and Axum (Rust). Each answers one /benchmark route with a short string.

The timings come from curl:

curl -w @curl-format.txt -o /dev/null -s URL

The analysis uses time_total, the time from sending the request until the whole response arrives. A small Go program in a container ran curl against the list of URLs and stored each result as CSV in an S3 bucket and in a database. With curl in a container the tests could run from several locations, and no browser affected the results.

The tests ran from two locations, one in a private cloud and one in the public cloud on Render’s cron service: every ten minutes during the 07, 09 and 11 o’clock hours from the private cloud, the 13, 15 and 17 o’clock hours from Render. Services were tested one at a time. I counted each run on the full hour as a cold start and the rest as warm starts, and took the minimum, maximum, median and mean of time_total for each.

Everything ran on free or the cheapest plans. The measurements took one week in the second half of 2023, and the prices here are from that week.

Results

Render gave the slowest responses in every round, with the worst over 300 seconds, and it emailed me several times that services were unavailable although nothing had changed. Overall Render had the weakest results and Railway was the clear winner. Railway’s median cold start, 0.26 s, was lower than its median warm start, 0.38 s, the one result I did not expect. For static websites Vercel had the lowest medians, 0.17 s cold and 0.16 s warm, and I stayed inside its free tier for the whole week, as I did on Render.

Table 1 of the thesis lists what Railway charged for each API over the test week:

Axum was the cheapest and also returned responses fastest. Serverless services bill for CPU and memory use, so I concluded that slower application code costs more in the long run.

Conclusion

The thesis concludes that Vercel is the most cost-effective platform for serving websites and Railway for web services. Serverless trades some application performance for lower costs of planning, deploying and maintaining infrastructure. For a feature a few people use each day that has to ship quickly and cheaply, it is close to ideal. Once the number of users becomes predictable, moving to private infrastructure pays off better than staying on serverless.