Since we added the dashboard we've had various requests from users to add two-factor authentication which is where you use an application local to you (such as on your computer or phone) to generate a one-time password which can be used in conjunction with your normal password to authenticate your logins.

The added security is obvious, if an attacker compromises both your API key and password they would still need to gain physical access to the device you run your authenticator on. Most people choose to use a mobile phone based authenticator for this reason.

So today we've enabled two-factor authentication for all accounts whether you're on our free or paid tiers you can benefit from the extra security two-factor provides. Below is a screenshot showing the new user interface within your dashboard for the feature.

We hope you like the apperance of the feature, we wanted to make sure it looks clean while being easy to understand and use. As the image informs you we're not limiting the feature to a specific authenticator. You're free to use any TOTP compatible authenticator which includes Googles, Authy, 1Password and more.

Thanks for reading and we hope everyone is having a great week!

Over the past couple of days we've been updating the stats tab on the Dashboard. Yesterday we added by customer request a country display to the recent detections section as shown below. We also added support for country information to the JSON output feature.

Today we've added a very neat looking interactive geographic heat map which shows the countries your positive detections are originating from. We're using maxmind's geoip data for this feature. This means we download their IP to Country database and do the IP Address lookups on our own server.

This is different to where we obtain our country/asn information for our main API and we are using maxmind here for speed so the graph can load very quickly. Below is an image of the map you'll find within your dashboard as of this post.

We hope you enjoy the new additions to the dashboard, these changes are the result of direct customer feedback.

Recently we've made some important changes we wanted to tell you about.

Network Improvement

The first is improving our network performance through better peering. As some of you may be aware if you've looked up our DNS records we use Cloudflare as our internet edge network partner.

This means before your requests reach our servers they first go through the Cloudflare infrastructure which is made up of 150 or so servers around the globe. Their goal is to have a server in every major city by the end of this year which would cover 95% of human civilisation.

As part of the Cloudflare service they have a paid sub-service called Argo which essentially funnels traffic between Cloudflare servers within a virtual private network over the lowest latency and highest bandwidth internet links available to Cloudflare.

When companies (like us) have Argo enabled it means your data first connects to a Cloudflare server closest to you, hopefully within the same city as you. And from there it only travels along Cloudflare servers around the world until it gets to one of our server nodes.

The end result is significantly lower latency for some connections. Argo is smart enough to determine if a connection will be faster through a native connection (You -> Cloudflare -> Plain Internet -> Us) or through Cloudflares Argo network (You -> Cloudflare -> Cloudflare -> Us).

So what are the benefits like? Well as you may be aware all of our three nodes are currently in Europe. We're planning to add one in North America and Asia but until we do so we're leaning on Argo to get faster peering for our customers that live outside of Europe. With Argo enabled we've been able to reduce our average Time To First Byte (TTFB) from 542ms to 382ms (29.52% improvement) which is a considerable difference when you consider we're handling millions of queries per day.

We love Cloudflare they're a great company that make a business like ours possible, we highly recommend them.

Customer Dashboard

The second change we wanted to discuss is something customers have been requesting for a while, the ability to update their card information without cancelling their current subscription. We're sorry it has taken us this long to implement this important feature but today we have made it available to all customers that hold a currently active subscription. Below are two screenshots showing what subscribed users see.

When clicking that update card details button you'll receive a simple model for you to enter your updated card details as shown below.

You may also notice that we altered the cancellation text to be a bit clearer. Now it explains when you request a manual change to your subscription tier we can prorate that change. So if you downgrade mid-way through your subscription we can refund you the difference and if you upgrade to a higher tier you'll receive a discount that takes into account how much you've already paid.

Thanks for reading!

In our last blog post we mentioned that we would soon start showing a message in your dashboard if you're still making queries on our v1 API. You may have already seen the message we're displaying but here is a screenshot of it.

As you can see in the message we have decided to turn off the v1 API on March 1st 2020. However if we still find a lot of traffic using the API at that time we will write some kind of basic translator which will forward queries to the v2 API and translate the responses back into the v1 format. If however very few queries are still being made towards the v1 endpoint we will remove it entirely on that date.

We think two years should give you all ample opportunity to upgrade, I'm sure many of you will have reason to visit the dashboard before the cut off point.

As we get closer to March 1st 2020 we will make the notice more visible, placing it nearer the top of the dashboard and changing the colours so it's even more visible and of course we will eventually send out emails to people using the v1 API when we're only 3-6 months away from the cut off date.

We hope that none of you will feel inconvenienced by the change, we want to make available the best API we possibly can and so setting a firm date allows us to evolve the v2 API without the need to keep making compatibility patches for the v1 API. Essentially giving us more time to work on forward thinking features.

Thanks for reading and have a great weekend.

Earlier today we launched the latest versions of our v2 and v1 API endpoints with new performance improvements and in the case of the v2 API new features too. We've updated both our Web Interface page and our API Documentation page to take full advantage of the new features. Below we've included some of the development process we went through.

In testing the average lookup time of a single IP with the inference engine turned off and VPN checks disabled has been reduced from 42ms to under 1ms. This is a huge decrease in database transaction latency which is due to our new on-server custom in-memory database program we wrote last month.

We did initially have some teething issues with the new database software in February which is why we took a very cautious approach to the rollout by spending many weeks gathering data and making changes to make sure the software was stable and consistent in its performance.

When dealing with sub-millisecond data retrieval from other programs you start to see variance in performance caused by the operating system kernel and we saw these performance fluctuations mostly on our weaker node ATLAS when our custom process and our web-server interacted.

It took some time to find out the cause (kernel pausing new application connections due to high processor load) and to create a solution that would scale from our 32 threaded monster node (PROMETHEUS) to our weakest four threaded node (ATLAS). We were able to solve the problem two weeks ago but we allowed the new API version to run under our development cluster for a few more weeks so we could gather more data and be sure the variance we found was solved by our changes which it happily was.

So that brings us to today, both the v2 and v1 API endpoints are utilising the new process. Next on our agenda is to improve VPN checks which are quickly becoming a large quantity of the lookups we perform due to the popularity of the VPN tag in queries. We've already added sophisticated multi-level caching to ASN lookups which has lead to a dramatic improvement in lookup speed there but there is more we can do for those kinds of lookups aswell.

We hope everyone enjoys the new features, as we said at the top of this post the API Documentation page has been updated with all the new flags, these are only included on the v2 API and we will soon be showing a message in your dashboard if your most recent query was to our v1 API. We're still going to be offering the v1 API for many years and updating it to maintain functionality but all new features (such as new flags) will only be included on the v2 API moving forward.

Thanks for reading and we hope everyone has a great weekend!

It has been a week since we last updated you on the progress of our new API update which contains new features and increased speed. So far the testing is going very well and we're currently filling the API's new database with data ready for live queries from our customers.

We expect to deploy the new API some time next week and we will be back-porting parts of the code to the v1 API endpoint, specifically the way it performs checks on IP Addresses and how it accesses our database of address data.

The new server process for handling database lookups has been working very well in testing, the performance has been extremely consistent with constant low data access times even under simulated loads from millions of incoming queries per second.

So that's where things are right now, we're on track for a release soon, the last thing I wanted to mention is we've had a few customers email us about the new API asking about how it will affect their use of the API so I just want to explain that for everyone.

This new API update does not change the response format you receive from our API in any way that should break your software. We're not changing the names of anything or how we present the information. We're simply making the results faster and adding more information to the results if you supply the flags to view such extended information. So once it's live you won't need to do anything except enjoy the increased performance.

Thanks for reading we hope everyone has a great week.

Over the past couple of weeks we've been working on upgrading the v2 API in a few specific ways.

1. Increasing result speed, lowering query latency for both single and multi-queries.
2. Giving you more control over the resolution of your queries
3. Giving you more information with your positive detection results

1. Increasing Speed

So to increase speed we've worked on this from a multitude of angles. Most of our queries are singular and so our goal was to dramatically decrease the time to process singular queries. Multi-checks are actually quite fast already due to our multi-stage cache priming that happens during the initial IP Address processing making subsequent address processing very fast.

So our goal has been, how do we get the same performance benefits that those primed caches deliver when we're only processing a single query. We think we've accomplished that by creating a new process on our servers within which we can place all of our proxy data in its most efficient data format.

We had already been using RAMDISK's to hold data but it was not held in its most efficient format and our RAMDISK had a lot of overhead for features we don't make use of. Our new custom process does away with all this and is designed specifically for our use case.

So what's the result of all this? Well in testing we've been able to reduce single query latency to around 1ms before network overhead. This is with VPN checks and the real-time inference engine turned off, both of which we'll be working to improve the performance of later.

We're still testing and benchmarking these changes but so far it looks promising that we will be able to change to the new code soon.

2. More Control

The second thing we've been working on is giving you more control over the resolution of your queries. What this means is, you can specify in days how close to the present time you want your results to be restricted to. For example perhaps only Proxies which have been seen as operating within the past 3 days as opposed to the past 7-15 days (the API's default maximum).

This new feature has been implemented as a flag called &day=# making it very easy to use. We're going to allow a resolution scale from 1 day to 60 days giving you great flexibility between very conservative detection which may miss some proxies and very liberal detection which may present some false positives.

3. More information

This is a combination of a few features. Firstly, because we're allowing you to specify the resolution in days of your detection results we're also going to give you the ability to see the last time we saw an IP Address operating as a proxy server. You'll be able to activate this feature by supplying the flag &seen=1 with your queries. We'll be displaying the result in both a human-readable "x time ago" format and a UNIX time stamp.

The other feature we're adding is the ability to view port numbers. This has been requested more times than we can count and it hasn't been something we've wanted to expose on the API because frankly it serves no security benefit but with how easy it is to scan an IP Address to discover running servers we've decided to implement the feature based on customer feedback. To activate this feature you'll be able to supply &port=1 as a query flag.

Below we've included a paste containing a query result provided by the new API version with both &seen=1 and &port=1 flags supplied.

{
    "node": "PROMETHEUS",
    "1.10.176.179": {
        "proxy": "yes",
        "type": "SOCKS",
        "port": "8080",
        "last seen human": "5 hours ago",
        "last seen unix": "1520297212"
    },
    "query time": "0.001s"
}

When are these changes coming?

We're still validating the new API, the changes we've made essentially constitute a brain transplant, almost all the code that actually performs the checking associated with a query has been completely gutted and replaced with a new more efficient system based around our new on-server caching process.

It's our hope to have the new v2 API updated later this month and we may even back-port parts of it to v1 which the majority of our customers are still using. We're also going to be overhauling the API Documentation page as we've grown the flags available considerably with these changes necessitating a redesign to the page layout.

Thanks for reading, we hope everyone is excited to try the new API, we're certainly excited to get it finished for you all to enjoy.

Today we've rolled out an update to our statistics gathering and processing code with some great improvements.

The first change reduces the individual pieces of customer stat information (for your dashboard stats page) we need to sync between our server nodes by 50% while retaining the same information as before, effectively we have reduced redundant stats. This change will have a measurable effect on how fast your stats are updated within your dashboard.

The second change is we're now performing query refunds when you send us multiple addresses to be checked and the query volume goes over the allotted 1,000 or the 90 second processing time window is exhausted.

So for example if you send us 1,500 addresses to be processed you will initially have 1,500 queries added to your accounts query total. But if we were only able to process 700 of those addresses your account will receive a refund of the remaining 800 queries that we didn't process. The refund is almost immediate.

We feel this change (refunding non-processed queries) is fairer to our customers as it doesn't cost us much computationally to not process data that is sent to our servers and it will allow you to re-send the data for processing without worrying about query overages.

Also before we made this change we never charged for invalid addresses, these new changes affect only multi-check queries that go above our query allowance of 1,000 addresses per query or queries which exhaust the 90 second processing time window.

During these upgrades we also discovered and corrected a bug which under-reported VPN, Proxy and Refused query breakdowns as displayed within your dashboard when you made use of the multi-ip checking feature (both via API and our Web Interface). This bug only ever affected our v2 API endpoint and only when performing multiple checks in a single query.

Thanks for reading and we hope everyone had a great weekend.

Today we've enhanced the Web Interface page with a new checkbox to enable ASN lookups on queries. This new feature will show you the ASN name, provider and country for the IP Addresses you're querying and it does so with a subtle hover tooltip as shown below.

We know you'll find this addition to the web interface useful as we ourselves make many manual queries to the API to get country and provider information for IP Addresses so this feature will be something we make heavy use of ourselves.

Around a week ago we changed the ASN lookup code on both our v1 and v2 API endpoints with the goal of speeding up these queries.

Due to the way ASN data has to be computed before it can be searched effectively it has always been resource intensive to perform ASN data lookups but with our new code we're now pre-processing all ASN data so it's in its most easily searched representation before any queries are made by our customers.

We've also changed how we're caching ASN data meaning repeated lookups even hours apart will have a much higher cache hit rate. All of our API cacheing functions are custom made and highly tuned utilising system memory.

The result of all this work has been significantly faster ASN results, we're now seeing around 80% of our ASN queries answered in under 55ms when previously 99% of ASN queries took over 300ms. We're also seeing a 30% query cache hit rate for our new level-1 ASN cache (IP to ASN) and 70% cache hit rates for our level-2 ASN cache (ASN to company and country).

Only around 4% of the queries we process every day utilise the ASN flag and that's likely due to either the added latency it creates or the customer not needing to perform ASN lookups. So whilst this is a small area of our business it was important for us to improve its performance which we have now done.

Another added benefit of this change is lower CPU utilisation on our cluster nodes which will allow us to answer all queries even faster than before and increase the volume of queries each node can process.

Thanks for reading and have a great day!