Recently, I started doing research into protocol buffers and decided to run performance tests to see if they are a viable option for reducing server response time over traditional JSON APIs. As I am new to protocol buffers, I made a demo site to test the different API types. See it here.
The demo site was made with the intention to test two factors:
- Will using protocol buffers make a website faster?
- If it does, is it fast enough to make the extra overhead worthwhile?
The github readme goes into detail about my testing philosophy, but in general the premise is as follows:
- One test will consist of a series of GET calls for both protocol buffers and JSON
- Each GET call will:
- produce a random string (of predetermined size) for all data keys included in the specific test;
- starting at 1 byte of data ;
- increase data by 50% with each following call;
- increase data to a maximum size (depending on test type);
- repeat when max data size has been reached for the alternate API type;
- calculate the time for each call from the start to finish of the GET call and when the response has usable data for the UI to consume (which includes decode time for protocol buffers)
The idea is to simulate various data sizes one would get from a similar GET call in an actual application.
To determine the results, the percent difference will be calculated between the different API calls for each data size.
- During one pass, the results of the protocol buffer API with 1 byte of data will be compared to the results of the JSON API with 1 byte of data.
- The percentage differences for all data sizes and for all passes will be averaged out to display an overall percent increase or decrease in the APIs performance.
- All results are represented using JSON as the baseline and comparing it to protocol buffer, so a percent decrease indicates protocol buffers were faster.
I tested this in Chrome, Firefox and Edge, and results varied in my tests between the different browsers and test types. Overall, there was a faster response time for protocol buffers (about 5-10%) for the regular message tests in both Chrome and Firefox. However, there was an increase across the board for the tall message tests.
Edge, by far, had the worst protocol buffer performance. I am not sure what to make of the significant increase in response time in Edge and will need to do some research into this anomaly.
For now, I’ll stick with what I do know. There is no denying that using a protocol buffers will decrease the amount of data being sent from the server. How much data is saved depends on the schema-to-data ratio. Looking over the result charts, it can be seen the data figure is significantly smaller when the majority of the data package are the schema keys. Since protocol buffers separate the schema and do not send them with every request, there is a significant drop in data the server needs to send. As the data within the data object grows, the schema keys become less significant and the decrease in the amount of data sent becomes obsolete.
In terms of developer usability, using protocol buffers was easy. There was an initial learning curve as is with every new technology, but the end result was only a few extra lines of code. I used protobuf.js libray. This library is well maintained and the owner is very active and was responsive to my inquiries when I ran into a stumbling block.
Depending on your data structure there is a case where it could improve speed. If your data structure consists of a large or complicated schema and the data within those keys are minimal, protocol buffers would be separating out a significant portion of the data. This decrease in data should allow for a quick enough response to compensate for the decode time on the frontend.
As for usability, I would say that despite the learning curve, the effort to implement the buffers is fairly simple and should not be viewed as a reason for not implementing it.
I’ll be continuing development on the demo project to add tests for different kinds of data structures and to see if I can more accurately determine the point of diminishing marginal returns.
Feel free to reach out with any suggestions or comments.