Introducing our new v2 API

Over the past couple months we've been working hard on a new version of our Proxy Checking API. And today we're proud to launch it officially and welcome you to try it out.

You may remember last month we shared with you a new API endpoint called v1b and at that time we weren't quite sure if we should launch the new API under the /v1/ endpoint because it created a lot of code bloat to make the new API compatible with our old result format whilst still being able to support the new features we've included.

New features such as:

  • Proper formatted URL strings so you can supply [ipaddress]?key= instead of [ipaddress]&key=.
  • The ability to send your IP Address to be checked using the POST method instead of just GET.
  • The ability to check up-to 100 IP Addresses in a single query using GET or POST methods.
  • The ability to disable real-time inference checking with a new inf flag.

v2 has all of these and to support the multi-checking feature we've had to alter our result format so that the proxy declarations are nested under the IP Addresses. We did (with v1b) create a compatibility layer so that if you were checking a single IP Address it presented the old format. But this created code bloat and frankly we didn't feel it was a good trade off.

So instead we're going to maintain our /v1/ endpoint for a long while (probably until 2020-2022 depending on usage). And in its place we will be presenting /v2/ as our main API endpoint from now on.

If you're worried about how this change will affect your code, don't be. The API is still very easy to query and in-fact in our own PHP Function (available on GitHub) we were able to upgrade with just two changes, the URL we were querying and the JSON conditional statement.

Essentially this: if ( $Decoded_JSON->proxy == "yes" && $Decoded_JSON->ip == $Visitor_IP ) {

Became this: if ( $Decoded_JSON->$Visitor_IP->proxy == "yes" ) {

It's just that simple. Now again you do not need to rush around and change your API over to v2, we will be supporting the v1 endpoint for many years yet. But if you want to get on the latest and greatest you're more than welcome to do so.

You'll find that we've updated our API Documentation page to include the new /v2/ endpoint and we've also spruced up the pages appearance. We hope you enjoy the new look.

We've got a lot of things coming in 2018 and this is the foundation on which we'll be building them. Thank you everyone for reading and we wish you a Happy New Year!

WordPress Plugin Now Available!

In our yearly retrospective we teased that a WordPress plugin was being made that would be available in early January. Well it was approved much faster than anticipated by the WordPress repository maintainers and so the plugin is available here on the WordPress site right now.

With the release of this new plugin we've also added a dedicated plugins page to the website where we'll maintain a listing of all the plugins that support the proxycheck.io API. As before with our Code Examples page if you've made a plugin let us know and we'll feature it just like we've done with this WordPress plugin made by Ricksterm.

We hope you all enjoy the new plugin and have a lovely new year.

A Giant Year

This year has been giant for us at proxycheck, we added lots of new features and dramatically overhauled our service, website and API. Below we'd like to share with you all the changes that happened this year.

Outreach

  • We started this company blog that you're currently reading
  • We started a GitHub account featuring PHP client code
  • We started a Twitter account and have been tweeting new features
  • We created a new contact us page with live chat support

API

  • We greatly decreased query latency through code refactoring
  • We added query tagging support
  • We built an Inference Engine to discover new proxies and to curate our existing data
  • We created 20 honeypots positioned around the world to capture malicious activity to further feed our Inference Engine
  • We vastly improved our VPN detection
  • We added support for VPN detection in IPv6 address ranges
  • We vastly improved our ASN flag support which now also supports IPv6 alongside IPv4

Website

  • Website gained a new look with drop-shadows, subtle animations and vibrant colours
  • A top navigation bar was added and tab bars were placed on some pages
  • We added a Pricing page and overhauled the Web Interface page
  • We completely remade the Service Status page
  • We made the website more mobile friendly with media queries
  • We remade the API Documentation page which now features dashboard API examples under a new tab
  • We significantly improved our Code Examples page with examples added for python, node.js, C# and Ruby
  • We added Stats, Whitelist and Blacklist features to the Customer Dashboard

Payments

  • We switched from one time yearly payments to monthly and yearly subscriptions (which you can cancel at any time)
  • We expanded our plan sizes into lower and higher priced plans (plans from just $1.99 a month all the way to $99 a month).

Infrastructure

  • We added a third server node to the cluster called ATLAS
  • We altered our international routing to enable lower latency access to our server nodes worldwide
  • We significantly upgraded our PROMETHEUS node going from 6 to 16 CPU cores

Email

  • We greatly improved the appearance of our emails and now bundle our CSS in the email itself for reliability
  • We standardised all our emails look and feel by creating a standard callable email function used by all our code
  • We send more emails to you for things like email/password changes and query overages, payment failures etc

With all these changes the service has become really fleshed out but we're not done. We still have features planned for next year including our new API that allows upto 100 IP Addresses to be checked in a single query.

We're also working on a batch processing and webhook system which we think will be very beneficial to some of our largest customers, this should be available some time early next year after the multi-check API is brought online.

The final thing we wanted to discuss is our free tier and how we've been distributing larger free plans to communities that need them most. Protecting websites, forums, game servers, payment gateways and more is something we're very proud to do and our service was actually started due to our founder needing this very service to protect their own online properties which included chat rooms, forums, websites and game servers.

That's why we've from the very start offered a generous 1,000 free queries to anyone that signs up, we have no intention to restrict that offering or to disable our premium features. We're aware that there are many competitors in this space that differentiate between free and paid customers by restricting features such as limiting Whitelist/Blacklists, not offering statistics or easy to access online support. We're different and proudly so.

We've also gone out of our way to give larger free plans ranging from 10,000 to 80,000 daily queries to many people who run free online games, chat communities, forums and support groups. We're proud to support forums that help teenagers and young adults with thoughts of self harm and also open source developers who release software we all benefit from.

This year has been really great for us. We've seen huge volumes of new customers and also many developers across the web integrating our API into their products and services, in-fact we have a WordPress plugin being developed on our behalf which should be available in January.

We'd like to thank everyone who has taken a chance on our service and we're really looking forward to bringing you new features and improvements next year and with that last sign off we also want to wish everyone a Merry Christmas and a Happy New Year.

New Dashboard Graph

The stats tab in the dashboard is the most heavily trafficked part of the site for registered users and that's because it's where you gather insight about the positive detections being made by our API on your properties. It's also where you can monitor your query allowance to make sure you're not going over your plans daily allotment of queries.

Today we're improving the stats tab with a new graph so that you can quickly see what your months been like without needing to page through each days queries. We're still giving you those full granular bar charts and the JSON API is still there for you to export and graph your queries however you wish but we've built in a really nice graph as shown below.

Our new graph is fully interactive so you can just hover your mouse over the data to see detailed number breakdowns and you can also toggle parts of the graphed data by clicking on the keys along the top. This is especially useful if only a small amount of your queries are positive detections which is common.

We know that you'll find the new graph useful as we're often asked by current customers what sized plan they should purchase based on their current usage. The new graph will make those decisions easier, especially as it shows your highest query days as large peaks and those are the days you want to plan ahead for.

This is likely the last major update we'll be releasing before the new year as we're winding down things for the Christmas break and new year celebrations. Thanks for reading and we hope everyone has a wonderful holiday and a happy new year.

Multi-check API update

Since we launched our multi-check API yesterday we've been hard at work improving performance and squashing bugs. Today we'd like to share with you some progress.

Firstly there were some bugs with the IPv6 VPN detection with regards to Google address spaces. This has been corrected in both the /v1b/ endpoint and in the back-ported code which is running on our main /v1/ endpoint.

The second IPv6 bug we had was with VPN detection. If you had not set the VPN flag to on but checked an IPv6 address it would be checked against our VPN data and a positive VPN result presented. This has also been corrected today in our /v1b/ and /v1/ endpoints.

The third bug we dealt with today was with dashboard statistics. Under certain circumstances you may have had a discrepancy in your total API queries reported at the top of your dashboard compared with the graphed breakdown of your query statistics. This was caused by underreporting on some negative detection scenarios. This bug only affected our /v1b/ endpoint.

Apart from fixing these bugs we've also corrected some very specific edge-case bugs. For example when performing a single IP check and the address entered into whitelisted IP ranges you may have received a response without the IP Address being repeated back to you in the JSON response. This behaviour has been corrected.

New functionality wise we've improved how we handle invalid IP Addresses. So previously you would simply receive a vague message indicating that one or more addresses were invalid but it didn't list the actual addresses you supplied. That's not an issue when you're checking a single address but with multi-checking you need to know which addresses in the data you've sent to be checked were invalid. To that end we now display that information back.

We're also planning to improve the 100 check per query limiter code. At present it simply stops processing your addresses once it reaches 100 and does not output to you a list of unchecked addresses. We'll be changing this behaviour soon to indicate which addresses were unprocessed due to hitting the limit. We'll also be changing statistic behaviour to account for this, at present if you send us 500 IP's you'll have 500 queries registered to your API Key even though we only processed the first 100.

So that's a quick update of where we're at. The new API is coming along steadily, we've already improved the performance since yesterday and we're squashing all the bugs we find as quickly as possible. We're on track for an early January 2018 rollout to our main API endpoint address.

Thanks for reading and have a great day.

Introducing IP Multichecking to our API

One of the biggest requested features from our customers over the past year has been the ability to check multiple IP Addresses within a single query. This feature has many benefits including reduced TLS handshake times, reduced resource usage from multiple webserver connections and decreased API latency through resource reuse.

To put it simply, it's a lot faster to perform one query with a 100 IP payload than it is to perform 100 queries with one IP payload each. We've tuned the new API for this multi-payload scenario and the performance improvement is dramatic as our benchmarks will show.

Before we get to those, we're considering this feature experimental. Due to this, the enhanced API is only accessible through /v1b/ with the b meaning beta. We're also supporting the submission of IP Addresses via GET and POST. You should use POST, the GET input is just there for testing in your browser.

We also want to be clear that this is not simply an abstraction endpoint that calls our API internally (and individually) we have literally gone through the API and rewrote every part of it to handle multiple checks. So this differs to how our web interface page has functioned (we will be transitioning that page to our new API soon).

So lets get to the benchmarks. Firstly we performed our testing multiple times with different addresses and averaged the results. There was not much deviation between tests. We did all our tests with 100 IP Addresses and we used TLS encryption.

IP Addresses that are NOT already in our data set with real-time Inference Engine turned ON

  • v1 (current) API with 100 queries each with 1 IP Address: "query time": "65.35s"
  • v1b (beta) API with 1 query containing 100 IP Addresses: "query time": "36.184s"

This is an impressive reduction, but watch what happens when we disable our real-time Inference Engine.

IP Addresses that are NOT already in our data set with real-time Inference Engine turned OFF

  • v1 (current) API with 100 queries each with 1 IP Address: "query time": "45.221s"
  • v1b (beta) API with 1 query containing 100 IP Addresses: "query time": "6.133s"

Now we're seeing a much larger decrease in query time. To be clear, our past Inference Engine cached data is still being processed here, so all past determinations made by the real-time and post-processing Inference Engine are still being utilised here but actual live determinations have been turned off.

Now finally lets take a look at positive detections. This is where the IP Addresses being tested (all 100) are already present in our data set but not within caches. So it's still searching all of our data but it's finding matches throughout our data set as opposed to never finding a positive detection like the tests above.

IP Addresses that ARE already in our data set with real-time Inference Engine turned ON or OFF

  • v1 (current) API with 100 queries each with 1 IP Address: "query time": "22.372s"
  • v1b (beta) API with 1 query containing 100 IP Addresses: "query time": "0.639s"

Here again we're seeing a huge decrease in query time. This is where we're seeing multiple TLS Handshakes and HTTP Connections being removed from the query overhead and our in-memory resource reuse has come into play.

So how do you start using the new API. We've made it really simple when you perform a query to the v1b API with multiple IP Addresses simply place multicheck in the IP field and then provide your multiple IP Addresses in a POST request called ips with each IP separated by a comma. If you want to use it in a GET request instead change the singular IP to multiple IP Addresses also separated by a comma. Below we've provided two examples.

GET request example

POST request example

You can still use your normal flags with these requests, for example ASN, VPN, Time, Node and we've introduced a new flag just for v1b called INF and as you probably can guess this disables our real-time Inference Engine so that you can perform multiple checks faster. To disable this feature provide &inf=0 in your request as by default it's turned on.

We're limiting multi-checking to 100 IP Addresses per query right now but we do intend to increase that limit after it comes out of beta. We hope you will all give it a good try and provide us some feedback which we welcome you to do at [email protected]

The last few things we wanted to mention about the new v1b endpoint is it does still support singular IP checks and the JSON result format is exactly the same as it has always been when perfoming a single IP check. You will see the new multi-check format only when performing multi-checks.

And finally since this is our new API we are working on it full time now. It has some added enhancements that our older API didn't have including better IPv6 support for VPN detection (since backported to /v1/ today) and we've moved around where certain checks are performed so you can now blacklist Google and Cloudflare IP Addresses, Ranges and ASN's from your dashboard and have those blacklists adhered to where as before they weren't.

Thank you for reading, we hope you're all having a great week and we look forward to hearing your feedback about this new feature.


Back