Today we have rolled out new versions of our v1 and v2 API's with a focus on reducing query processing time and the improvement is quite drastic.

Prior to today it was common to have a full query which means having all flags enabled (VPN, ASN, Inference etc) take between 7ms and 14ms to process. Today we've been able to reduce that to between 1ms and 3ms on average.

The way we've been able to accomplish this is by porting some of our changes that we've created for our yet unannounced v3 API back to v1 and v2. The changes are mostly structural dealing with the processing of our code, how it's compiled and cached between queries and finally how it's executed again for subsequent queries.

Although we've always reused processes for different queries (due to the time it takes to setup a new process being too long) we're now doing it more efficiently with more data being retained by these processes so they don't need to reload as much information into memory between queries.

We've also altered our caching system for our code to have a tiered storage approach. Code files are now loaded from disk into memory by one of our processes and then the opcache retrieves the code from that memory based file cache before compiling it and storing the subsequent opcode in memory. This results in more consistent performance as the opcache no longer needs to reload or check modified dates of code files from our physical disks, they are instead held in memory by our file based caching process.

This change is important because opcache checks code files frequently to determine if the file containing our code needs to be re-compiled and cached again, storing the files in system memory thus keeps performance consistently high over long periods of time.

We're also making more efficient use of the compiled versions of our code by removing comments and other nonessential text from the code files prior to compilation which makes the compiled versions smaller and faster to run. Finally in the code itself we've reduced database calls which cause processor context switching so we waste fewer CPU cycles through gathering and sorting information and more time displaying information. This ties into the change we mentioned above with regards to our re-used processes storing more data in memory for each query to make use of.

So what is the net benefit of having the API respond this quickly? ~7ms to ~3ms may not seem like a lot of time to save. But put simply, it allows us to handle 2.3 times more queries per second on each cluster node.

It also means we can answer your specific queries faster so you're not waiting as long for an answer from us, that allows you to use the API in more time sensitive deployments. Some of our customers are already so physically close to our infrastructure they can receive an answer (including network overhead) in under 20ms so we're getting to the point where every single millisecond we save counts.

Thanks for reading and have a great day!