HTTP vs Websockets: A performance comparison
In many web applications, websockets are used to push messages to a client for real-time updates. One of the more interesting and often overlooked features is that most websocket libraries also support directly responding to websocket messages from a client (acknowledgements in message queue-speak). The following example uses Socket.io to increases a global counter and return the new value to the client that sent the message:
Feathers uses this for exposing its APIs both ways, via a traditional HTTP REST API and completely through websockets in which case it also sends real-time updates. It allows to use either transport or both at the same time. Since it is just a very small and fast abstraction that wraps Express and Socket.io it also does not add a lot of overhead and performs about the same as implementing the routes and websocket handlers by hand (but without having to write all the repetitive glue code).
Usually we recommend using a websocket connection when getting started with Feathers because you get real-time updates for free and it is faster than a traditional HTTP connection. Until now I never had any real numbers to put behind the “faster” claim so it was time to do some real benchmarks. This post compares the performance of request/responses between HTTP and an equivalent websocket call. You can find the repository with the server, test website and benchmark scripts at daffl/websockets-vs-http. For a more detailed analysis of real-time message formats also see A comparison between WebSockets, server-sent events, and polling by Alexis Abril.
The server used for testing is pretty much the same as the example on the feathersjs.com homepage. It has one simple service that implements the method and replies with the id it got passed. This means a request to will return a JSON object like . The equivalent websocket message is .
To get a more realistic environment the server is hosted on Heroku using a 1x standard Dyno. The benchmarks are run on a 2014 Macbook pro with 2.8 Ghz i7 and 16 Gigabytes of RAM in the latest Chrome browser.
For the browser I set up a small web page that allows to run and time a fixed number of (parallel) requests and a (sequential) series of requests within a certain timeframe.
All browser tests do not include the time (~190ms) it takes to establish websocket connection. The times for a single HTTP and equivalent websocket request look like this:
On average a single HTTP request took about 107ms and a Socket.io request 83ms. For a larger number of parallel requests things started to look quite different. 50 requests via Socket.io took ~180ms while completing the same number of HTTP requests took around 5 seconds. Overall HTTP allowed to complete about 10 requests per second while Socket.io could handle almost 4000 requests in the same time. The main reason for this large difference is that the browser limits the number of concurrent HTTP connections (6 by default in Chrome), while there is no limitation how many messages a websocket connection can send or receive. We will look at some benchmarks without that restriction a little later.
Another interesting number to look at was the amount of data transferred between both protocols. Once established, a websocket connection does not have to send headers with its messages so we can expect the total data transfer per message to be less than an equivalent HTTP request. Establishing a Socket.io connection takes 1 HTTP request (~230 bytes) and one 86 byte websocket frame. Excluding this initial connection setup, the data transfer for actual requests looked like this:
One HTTP request and response took a total of 282 bytes while the request and response websocket frames weighed in at a total of 54 bytes (31 bytes for the request message and 24 bytes for the response). This difference will be less significant for larger payloads however since the HTTP header size doesn’t change.
Now that we got some insight how a single browser client behaves and how much data is being transferred I was also curious about load tests from multiple clients and ran some benchmarks. For benchmarking HTTP requests I used Autocannon and for websockets I settled on websocket-bench which has similar options and good support for Socket.io. The only common data point both tools supported though was the total runtime of the benchmark, which is what we will compare here by charting the total (fixed) number of requests (per connection) over the time it took to make them (as requests per second). The benchmarks are each running 100 concurrent connections and, unlike the browser numbers above, include the time it takes to establish the websocket connection. Here are the results for 1, 10 and 50 requests per connection:
As we can see, making a single request per connection is about 50% slower using Socket.io since the connection has to be established first. This overhead is smaller but still noticeable for ten requests. At 50 requests from the same connection, Socket.io is already 50% faster. To get a better idea of the peak throughput I ran the same benchmark with a larger number (500, 1000 and 2000) of requests per connection:
Here we can see that the HTTP benchmark peaks at about~950 requests per second while Socket.io serves about ~3900 requests per second.
An important thing to note is that even when used via websockets, the communication with the Feathers server is still RESTful. Although most often used in the context of HTTP, Representational State Transfer (REST) is an architectural design pattern and not a transport protocol. The HTTP protocol is just one implementation of the REST architecture. What this means for web and real-time APIs in general is the topic for another post.
When it comes to scalability, one advantage of a REST architecture is statelessness which means that any server can handle any request and there is no need to synchronize any shared state other than the database. Feathers applies the same concept to its websocket connections. The only information attached to the socket is the authenticated user (to decide what real-time events to send). As long as real-time events are synchronized across multiple instances, this architecture can be scaled across multiple servers similar to a traditional HTTP setup.
The benchmarks I ran are by no means a comprehensive analysis of all the nuances in the performance difference between HTTP and websockets. It did however confirm my initial impression that for many cases websockets can be faster than a traditional HTTP API.
Enabling different communication protocols and being able to transparently switch between them without having to change your application logic was one of the key design goals of Feathers. There is nothing wrong with web frameworks that help handling HTTP requests and responses with newer language features, different design patterns or that are simply faster. However, I still believe that a protocol independent architecture and being able to dynamically choose the most appropriate transport protocol will be crucial for the future of connected APIs and applications. In the case of our benchmark, we were able to get a 400% performance boost by using a different protocol without having to change anything in our actual application.
Comparing Meteor.js and the MEAN stack
I started writing this article as an answer on Quora, seeing how there wasn't really any good comparison between Meteor.js and the MEAN stack (MongoDB, Express, Angular, Node.js).
First off, I should mention there's no such thing as "the" MEAN stack or framework. The acronym (explained above) generically identifies using these technologies in combination. There are several implementations of the MEAN stack (a community fragmentation problem in itself). Meteor is actually a mature full-stack real-time web application platform (a framework plus a lot of utilities) that has been developed by a very well-funded and highly knowledgeable team since 2011 and has a very large community.
What Meteor gives you for free
Meteor gives you a lot more out of the box. The client and the server communicate data updates seamlessly and automatically, without you having to write any boilerplate data sync code.
The MEAN stack is just MongoDB, Express, Angular and Node.js bundled together, but there's nothing seamless about it. You have to do all the wiring-up yourself between MongoDB and Node.js, between Express and Angular, create REST endpoints and consume them etc. - all this just to get a basic web app going, without any features that Meteor gives you for free: hot code reload, mobile development (web, Android and iOS apps from the same code base), reactive templates that update automatically when data on the server changes (try to write code for that manually and make it run correctly over intermittent network connections, and make sure it's secure), session management, packages that can install components both on the server and on the client (e.g. server-side autocomplete - you won't find this anywhere else; Twitter's Typeahead and the like are client-only solutions).
With the MEAN stack, when you make a REST request for a table/collection, you're essentially saying "Send me the results of this query". With Meteor, you subscribe to the results of that query, and any newly created or modified documents that matched will be automatically sent to the client over a fast WebSocket connection.
Thanks to its isomorphic APIs (the same methods work on the client and the server, e.g. ), Meteor makes it easier for one developer to build an entire full-stack app, or for a team to have a better understanding of the code base across the project. The MEAN stack adds to the separation between the server and the client due to different APIs being used.
No callback hell
Meteor's server side runs on top of Node.js but eliminates callback hell by using Fibers. Even on the client, you can avoid callbacks and even promises by using reactivity via the simpler Tracker and Reactive Method packages.
In Meteor, your server code runs in a single thread per request, not in the asynchronous callback style typical of Node. We find the linear execution model a better fit for the typical server code in a Meteor application.
7 Principles of Rich Web Applications
Guillermo Rauch, the well-known author of Socket.IO, has written an excellent article call "7 Principles of Rich Web Applications". He's also delivered a talk at BrazilJS about the principles:
- Server rendered pages are not optional
- Act immediately on user input
- React to data changes
- Control the data exchange with the server
- Don't break history, enhance it
- Push code updates
- Predict behavior
Meteor implements #2, #3, #4 and #6 for the developer. #1 (server-rendered pages) and #5 (enhance history) are handled by ecosystem packages (Spiderable, Server-Side Rendering, and Iron Router). (#7 as described by Rauch, isn't really in the purview of full-stack frameworks.) By comparison, the MEAN stack needs to be coupled with Socket.IO to even have the pieces to implement these principles - Meteor already has them all glued together.
Meteor also has the entire toolchain built-in, from compiling Coffeescript automatically to minifying and concatenating CSS and JS. Within seconds from saving a file, all client web browsers will automatically reload and re-render the app thanks to hot code push (#6 above). The UI state (form inputs, scroll position, selection) is preserved (another fun thing to try to get right with the MEAN stack).
Such as extremely convenient packages - one of the "Eureka" moments with Meteor is when authentication with user/password, Google, Facebook and Twitter is added just by including a package for each OAuth provider and one line of template code to indicate where to place the login widget:
Meteor has over 4,3000 packages. MEAN.JS has only one module so far, for SEO. Meteor lets you deploy with one command to free hosting at yourapp.meteor.com. MEAN.io doesn't support deploying yet.
Which is easier to learn?
But isn't that a lot to take in all at once? Actually, not. Thanks to Meteor's focus on simplicity (one of its Seven core principles), Meteor in its entirety is far easier to learn than Angular alone. Many have found Angular to be a pain to learn and ended up frustrated - see An Unconventional Review of AngularJS:
Angular is... band-aids over self-inflicted wounds
You can get the basics on Meteor in just one hour - head over to http://meteor.com/try. Meteor's documentation is also very well-written. By comparison, the documentation for the MEAN.io stack is in such a shape that today I went through the Packages section and found an embarrassing number of typos in just a few minutes, all within two pages of the documentation. Packages seem to have no documentation, either. And not to discredit the authors, but you can't easily find who the core team is - the link in Credits is broken. The documentation for the alternative, the MEAN.JS fork, starts with,
After downloading the source code go over the official documentation of MongoDB, ExpressJS, AngularJS, and Node.js, then continue with the MEAN.JS Documentation.
Given that it really takes only one hour to make something useful with Meteor, from scratch, it's worth simply giving it a try first, to have a reference before starting to learn MEAN.io or MEAN.JS.
Of course, you'll still need to learn MongoDB well for a production app, but Meteor doesn't send you there to get started. The 80% most common cases of using Mongo are already explained in the Meteor documentation.
Meteor also saves you from having to research what libraries to use for the various layers of the application (transport, data synchronization, API, security, CRUD operations, templating etc.). All the pieces that make up Meteor function together very well.
By contrast, when considering the MEAN stack, you start with the choice between MEAN.io and MEAN.JS. The short story is that the main MEAN.io contributor, Amos Haviv, Forked out of an open-source conflict in January 2014. You can read more about the story and differences on StackOverflow. This has split the community into ~6k GitHub stars for .io and ~2k for .js. Meteor by comparison has 23,000 GitHub stars.
On StackOverflow, right now there are over 10,000 questions tagged meteor, and about 1,100 total tagged mean, mean.io and meanjs
More more details on the community size, see the Community section in Why Meteor.
When not to use Meteor
As with any tool, Meteor is not always the right solution. I've expanded on when not to use Meteor in the corresponding section of my Why Meteor article (e.g. when you're building a client-only app, or when you require interfacing with databases other than MongoDB). To compare apples to apples, have in mind that the MEAN stack includes a server and MongoDB. So let's see when a MEAN stack might be a better choice for a new web application:
- If you already have most of the pieces of your app written using the MEAN stack. Meteor provides many extra niceties, but make sure they justify the migration cost.
- If you're only building a web service/REST API and don't need any client functionality. Meteor is a client-server platform. (Side note: if you're considering REST, consider GraphQL instead)
- If you don't really need your web app to be real-time. There is a RAM and CPU cost for providing real-time data sync between the server and subscribed clients, even though one $5/mo low-specs DigitalOcean server can handle a basic Meteor app.
- If you're building a website, rather than a web application, and it's more important to deliver the first page as fast as possible, rather than be able to prototype or develop very fast
- If you already need to scale your app to millions of users right now - though with Meteor 1.3, two new developments will help in this area:
- The Galaxy scalable hosting platform
- Meteor's Livedata system now supports tunable queries to help scale large apps.
I've found that Meteor saved me from a lot of analysis paralysis, besides making me a ton more productive than if I had to put together the pieces in the MEAN stack. My suggestion is to read a bit on Why Meteor could be a good choice or not, give it a try first for one hour (meteor.com/try), then look at the MEAN stacks, and see which would work best for your project.
PS: skip to 44:30 in Guillermo's talk above. Someone asks him about Meteor. A month later:
love [Meteor] too <3
Showing changes from previous revision. |
by Dan Dascalescu, 2017-02-15