If you've been reading our blog for the past few years you may have seen a post we made in December 2019 where we detailed our inference engine called Raven. This is the software we created that runs not only on each of our nodes for real-time inference but also on separate dedicated hardware tailored specifically for post-processing inference.

Since that post we've changed where and how Raven functions. We broke up our single dedicated inference server (STYX) into many separate servers and repurposed STYX as a distributor of work instead of a processor for Raven. We had to do this due to the service becoming so popular we couldn't process the volume of addresses we were receiving in a time frame that made sense.

This month we've been hard at work improving Raven and the associated infrastructure that supports it. We've reached a scale where traditional databases, storage systems and networking are not scaling for our use case, we want to be able to process many more addresses per second and in a more thorough way which requires more resources at every link in the chain from the way addresses are collected, transported through our infrastructure, processed and delivered back to our cluster nodes.

To this end we've completely changed how addresses are collected from our cluster, it's now multithreaded and scales seamlessly to the volume of data waiting to be picked up. We're also now storing addresses in a high-performance in-memory database served by MariaDB. We're seeing very high transaction throughput combined with extremely low CPU utilisation from MariaDB and in-fact this one change from our prior custom solution reduced CPU usage from 97% to 30% on our work distribution server.

But that's not all, Raven for us is more than just a data analysis tool, it also includes what we call agents which allow it to be extended with plugins that serve as data collectors and data formatters. Essentially a way to feed Raven auxiliary data through a multitude of means. For instance processing firewall logs from our data partners or even agents that probe addresses directly to see if they're running proxy servers.

That last agent we mentioned that probes addresses directly has become a very important tool for Raven because it provides conclusive evidence which helps to reinforce its prior conclusions and thus help it to make better decisions in the future. Another advantage of this particular agent is its ability to find new proxies from where we have no data. This is important because we, like all anti-proxy services, operate a network of scrapers which scour websites that publish proxy and vpn addresses in an attempt to collect as much data about bad addresses as possible.

The problem however is many of these websites have data that overlaps with one another and so there is not many sites publishing proxies that we don't already know about. We spend a lot of time locating new sites and often even if they list thousands of addresses as being seen within the past several minutes we already detect 99.9% to 100% of them. So the ability to seek out unique addresses that have never been published publicly is important if we want to have a full picture which is certainly our goal.

And indeed we do find many unique proxies on our own, in-fact we find hundreds of unique proxies daily that have never and in some cases will never be listed on publicly accessible proxy indexing websites. With how important this agent is to our service we spent the last few days rewriting it to be faster and smarter. We've come up with some subnet searching algorithms that increase the chances of finding bad addresses without needing to scan an entire service providers address range in addition to some other improvements that we're going to keep close to our chest for now due to their trade secret value.

The last piece of the puzzle has been iterating on Ravens inference models. In the past we would collect a subset of important decisions and their outcomes to train Raven. It would actually almost take a month each time. But we've been able to improve the training time by breaking up the data into smaller units which can be iterated on across different computers. In addition to that we upgraded our main workstation that we would traditionally compute these models on which has cut the training time in half. We're now able to produce a new model in 8 days down from the 26 days it took previously which is a significant improvement that allows us to tweak Raven more often.

So that's what we wanted to share with you today. If you often monitor our threats page which is where we post unique proxies we've found that haven't been seen on indexing websites before you may notice a vast increase in the postings over the past 2 days. This is going to continue to ramp up as we further tweak the new software and find the right balance between detection rate and processing throughput.

Thanks for reading and have a great week!

In 2018 we introduced a changelog feature to our website pages so we could easily convey changes to you. This has not only been a big hit but it has garnered us some praise because we deployed it on more than just feature pages like the Customer Dashboard but also on our pricing, privacy, terms of service and GDPR pages.

We did this because it adds transparency. When there is a ledger that details what was altered you can monitor and question us about the changes. We feel this feature is very important which is why today we've enhanced it to make it both easier to read and faster to browse.

Above is the new changelog v2 interface which is now live across all our pages and as you can see it's colour coded. We now list new features in green, changes to existing features in blue and fixes in purple. We've also added some icons to the side which are more for fun than functionality. The new interface uses our updated interface guidelines that we've been adjusting the site to meet over the past several months.

With this transition we did go through every log entry so far which took considerable time. The entries are now catagorised and stored in a more convenient format so we can update things in the future with less effort, we may even make a global page to view all change logs in the future. With this change none of the prior entries were removed so you can still scroll all the way back to 2017 if you're so inclined.

Thanks for reading and have a wonderful day.

Today we've added a new cluster node called Nyx to serve the North American market alongside Cronus and Metis. If you read our earlier blog post here you'll know we recently began our infrastructure expansion outside of Europe by adding servers in North America.

The aim has been to lower customer access latency by reducing the distances between our customers servers and our own servers while also maximising service availability in the event of network disruption or hardware failure.

Over the past several months we've seen our daily API request volume increase substantially and the North American region especially has seen a huge increase in just the past 30 days which has been driven by an influx of new customers and even existing customers deploying our service to their pre-existing servers in North America after we delivered the lower access latency they required.

Due to this increase in NA based requests we needed to increase capacity and that's why we've added Nyx today. We're still looking for some great infrastructure in Asia which will be the next region we expand into, we're seeing heavy traffic from Japan, South Korea, Singapore and Malaysia which we want to serve better and we hopefully will later this year.

We have been asked if we're still using bare-metal servers or if we've moved some of the cluster to cloud instances. We can confirm that we're still using bare metal servers, we continue to believe the highest security possible remains with bare metal and so that's what we're using for all cluster nodes and in-fact any of our servers that hold any kind of customer data or metadata are bare metal and wholly operated by us directly which is to say we don't use managed services from a third party.

Remember to follow us on Twitter for all the latest updates, we don't tweet often but when we do it's always worth the wait!

Thanks for reading and have a great day.

Today we've updated the customer dashboard to add a custom rules library which as of this writing contains 18 pre-made rules for the most common scenarios we see rules being used for. As time goes on we'll add more rules to the library as we've made it fully extensible and new rules can be easily created and exported to the library by our staff.

Below is a screenshot showing the library interface and as you can see it's very simple with categories along the left, descriptions in the middle and buttons along the right to add specific rules to your account.

The part that makes this new feature powerful is the fact the rules are editable by you. Simply add one of our rules from the library and then you can modify it just as if you had created it yourself thus allowing you to quickly get started by using the library as a templating tool.

This is a feature from our roadmap that we've been planning since the custom rules feature was first unveiled to you in mid 2019 and we're very pleased to introduce it today.

We've prototyped many different approaches to creating an easier to use rule system including walk-through guides and even a questionaire based wizard but ultimately we settled on this library system because it's the easiest to use and teaches you how to use our advanced rule interface through working examples.

Thanks for reading and welcome to 2021!

At the end of each year we like to look back and discuss some of the significant things that happened and this year we did have a few significant events occur.

Firstly we need to discuss the elephant in the room. The pandemic has been a constant presence since very early in the year and it shaped some of our decision making. We added four new nodes this year due to increased customer demand and part of this increased demand was due to the pandemic changing peoples online behaviour.

That is to say as people spent more of their time online the services they use turned to us for protection to a higher degree. We saw record numbers of customers increase their plan sizes, record numbers of new signups and record levels of conversions from free to paid plans.

Our service more than doubled in size this year by user signup and daily usage metrics which is significant for a service in its 4th year. This growth is somewhat bittersweet because the pandemic which has harmed so many is partly why we've grown so rapidly this year.

We had to disable our live chat support for most of this year due in part to the high volume of chats it was generating but also our support staff being less available because of the pandemic. This is a feature we do intend to bring back in 2021.

When we look past these aspects of 2020 we did launch a lot of new features and enhancements to previous features. One of the biggest new features we launched was Cross-Origin Resource Sharing (CORS) which we recently updated with enhanced functionality.

This features popularity took us a bit by surprise. We knew customers wanted it as they had mentioned client-based implementations to us often even if they didn't know how to articulate it exactly. Since we launched CORS many thousands of websites have added our service to themselves through this method which again has been a surprise. It is by far the most popular feature we've added since the service started with custom rules being a close second.

This year did also see us add extra information to the API including Continents and Regions. These were added not just to improve the usefulness of the API in your apps but also to assist customers who wanted to make more targeted custom rules within their accounts.

Another popular feature has been burst tokens. We added this to help customers handle those momentary spikes of traffic that can happen from an attack, going viral or launching a new product where purchasing a higher paid plan doesn't quite make sense yet. We've had a lot of feedback about this feature and customers really seem to love the peace of mind it brings them, we've solved what we're calling query anxiety.

One of the more recent developments has been our regional expansion. Adding servers to North America had been on our roadmap for over a year but we always had a lot of concerns about data security. When you use our service you entrust us with your data and we didn't like the idea of that sitting on servers in the United States, specifically EU customer data on US servers. To combat this "issue" we devised an internal data access and permission system that keeps your data accessible only to our regions that you're actually using.

We've always stored customer data at rest in an encrypted container format within our block based database system but all our servers always had the keys to decrypt any of that data. With the advent of our regional nodes this has changed and each region will only have the ability to decrypt specific account data (Stats, CORS, White/Black lists, Custom Rules etc) after the customer tied to that account performs an action that would necessitate accessing their data. These keys also expire regularly to revoke access when it makes sense.

Looking towards the future it is our intention to add nodes in Asia next year which will lower access latency to our API for that region like we did for North America. We also have traditionally done price alterations in January but we're not doing that this time. We feel the pricing we have right now is great and so no price rises are currently necessary.

We know this year has been marred by disappointment due to the pandemic but we will return to normality. It's our sincerest wish that all of you are safe and healthy as we enter into the new year.

Thanks for reading!

We're sure if you've been following our blog this year you would be surprised to see another new node announcement quite this soon let alone two. But it's true, we're adding two new nodes today called Cronus & Metis and these are quite special because they're the first nodes we're activating outside of Europe.

This year has seen us increase our capacity quite a lot to meet the growing demands of our customers and while we were intending to add new servers early next year we've pushed up our timetable because we're seeing increased request volumes from outside Europe.

Specifically 1/4th to 1/3rd of our traffic (depending on the time of day) now originates from America and Canada. Having these requests traverse the Atlantic Ocean to our servers in Europe has meant our north American customers are facing higher than acceptable latency and so today we've added two new server nodes in Canada just on the border with the United States.

Cronus was the greek god of time and so it's aptly named as its only job will be to serve the North American market with an aim to reduce their access latency to our API and Metis is the personification of prudence or in other more common language, cautiousness. And we're being cautious with our North American rollout by adding two servers for load balancing.

In addition to the new nodes we've spruced up the status page a bit breaking out where our server nodes are available. At current that is Western Europe, Eastern Europe and now North America. It is our intention to add servers in the Oceania and Asia regions to serve those areas in the same way and we will likely add such server nodes next year.

Like all our other servers these new ones are part of our unified cluster architecture and so while all North American traffic will go to Cronus & Metis it will seamlessly failover to our European servers if there are any problems. Your data is synchronised between all nodes and protected from downtime without you needing to do anything.

So that's what we have for you today and we hope you enjoy this one last present before Christmas.

Happy holidays everyone!

Today we've released some major updates to the CORS (Cross-Origin Resource Sharing) feature found within your dashboard and we're excited to tell you about them.

Firstly we've made some under the hood changes to how your origins are stored on our servers and processed by our v2 API endpoint which should reduce retrieval time from our database and lower the latency incurred when answering a CORS based request from customers, especially for those of you with a large number of origins on your accounts.

Secondly we've improved the import/export experience within the Dashboard. The exported CORS files will be easier to parse and edit with UUID's scrapped from the process, only domains are present within the exported files now.

Thirdly we've added wildcard support which means if you have a lot of subdomains you no longer need to enter them all manually and can instead put a star to indicate all subdomains and the main domain should be allowed to use CORS for your account. (example: login.site.com can become *.site.com).

Fourth we've finally added a Dashboard API endpoint (currently in beta but accessible to all customers) which allows you to list, add, set, remove and clear your origins but crucially it allows for large batch changes to be performed for both adding and removing origins which supports usage at scale. You can view all the documentation for this here

So those are the four changes to CORS, we know you'll find them useful, especially the API and the wildcard support which have been often requested by customers. One last thing, if you intend to use the CORS API please report any issues you come across to us and we'll work to remedy them quickly.

Thanks for reading and have a great weekend!

Today we've launched a major new feature called Burst Tokens which allow our customers to make even greater use of their plans without needing to lift a finger.

For a long time we've had customers coming to us with a simple problem. Most of the time their usage fits within their plan size but sometimes they have bursts of activity which go beyond their plan size. This is a problem because it doesn't make economic sense to increase your plan size just for those one or two days a month when you need a few more queries.

This scenario plays out fairly often especially with websites that receive unexpected viral traffic and game servers which are often targeted by DDoS attacks from disgruntled players.

So to solve this problem for our customers we've introduced Burst Tokens which while active allow you to go over your daily allowance by five times until the next daily reset time. And best of all the tokens are redeemed on your behalf automatically when you go over your daily allowance.

You'll receive tokens on the 1st of every month and only a single token can be consumed each day with the plan you have dictating how many tokens you're granted. For our free customers on our 1,000 daily query plan they are given one token to use each month while our Starter plans get 3, our Pro plans (as illustrated above) get 5, our Business plans get 6 and our Enterprise plans get 7.

As we said above you can go over your daily allowance by 5 times when a token is consumed. So if you're on our first paid tier which is our 10,000 daily query plan and you happen to go over that, a token will be automatically redeemed and your daily allowance for the remainder of that day will be 50,000 queries.

At this point you're probably wondering if this is a new paid feature and actually it's not. All past, present and future customers with an account will have access to the new token feature and in-fact by the time you're reading this you'll be able to see your available tokens in your customer dashboard. We've also updated our usage dashboard API endpoint with burst token availability.

And so that's all there is to it, a free upgrade on us to help supercharge the plan you already have. But don't worry, we'll still send you normal usage emails when you go over your plans daily allowance but they'll now also detail if a burst token was used so you know if it's time to upgrade your plan or if it's just a spike in usage that your tokens can handle.

With the launch of this feature we have released a new version of our API v2 dated November 17th. If you already have your version set to use the latest stable API version you will be using this version of the API automatically, otherwise you can select it within the customer dashboard. We're not expecting any implementation breakages but some of the status code messages have changed wording to indicate if a burst token has been consumed or not.

If you have any questions about the new feature as always contact us, we love to hear from you.

Thanks for reading, stay safe and have a wonderful day!

It's hard to believe only 9 months have passed since we introduced our Eos server node and yet we're already introducing another new server node to our cluster.

This year has been filled with difficulties as the world continues to grapple with the COVID-19 pandemic. A result of which has meant more people than ever before have turned to the internet for their communication with loved ones, entertainment, education and work.

As our company helps individuals and businesses protect their infrastructure we too have seen the demand for our services grow. In-fact we broke every record we held this year. Monthly, weekly and daily signup records to the service were easily broken multiple times as were our daily query volume records. We saw record levels of user activity on the website and general enquiries about the service from potential customers increased by an unbelievable volume.

And this is why it's so important to always be continually investing in our infrastructure. The previous blog post to this one explained how we had added multiple high-end servers for post-processing inference so that our proxy detection can continue to be the best available. Today we continue that focus by adding a new high performance server node to our cluster.

Aura is the Titan goddess of the breeze and fresh cool air of the early morning. And it is also now our most powerful server node featuring a high performance AMD Zen2 processor. This is the beginning of a new platform for us, this single server is the equivalent of three of our 1st generation server nodes in raw compute power giving us enormous growing capacity.

It is our intention to replace all our 1st and 2nd generation infrastructure with nodes of this capability and to keep the cluster around 10 servers or less spread out around the globe offering us redundancy against not only individual system failure but also geographic problems such as international fiber optic cable damage. Already we make use of multiple datacenters spread across Europe and we will expand on this as we add more systems to the cluster.

At the moment Aura is in the final stages of provisioning where we perform rigorous tests to make sure it's up to our standards. So far it's looking good and we're expecting Aura to answer its first customer queries starting tomorrow.

Thanks for reading, stay safe and have a great day!

Today we would like to share with you some updates we have regarding our machine learning infrastructure geared towards post-processing. This is where you send us an IP Address to be checked and after we give you an immediate answer we put it into a large pool of addresses to be examined where time is no longer an issue.

In February 2019 we made a blog post about a new server we introduced called STYX which was designed to do all post-processing inference to free up resources on our core cluster so they could spend more time answering queries instead of processing data.

You can see above a graphic we shared within that post illustrating how our (at the time) three cluster nodes would feed data into STYX to be processed by its many processing cores.

Since then the volume of addresses we process every day has increased to an unimaginable amount. To keep up with this growth we've increased our cluster size from 3 to 5 servers, replaced our weakest servers with stronger ones and gone to extreme levels of code optimisation all of which has allowed for our level of growth without spending obscene amounts of money on cloud providers.

But coming back to STYX we did hit a problem there. No amount of code optimisation can get around the fact that there are simply too many addresses to process on one system. We put in some stop-gap measures by creating a ratio system where only half of addresses were tested, then 1/3rd, 1/4th and finally only 1/5th. Eventually if we continued in this manner only a tenth of all addresses would be able to be processed by the post-processing engine on STYX.

And so that brings us to todays post where we have invested in an entirely new range of infrastructure dedicated to inference. They consist of various servers with various core counts. Some of the largest servers we've acquired for this now feature dual 18-Core XEON's. In-fact our inference infrastructure is now several times more powerful than our cluster that answers customer queries.

STYX is still with us but it has been repurposed as a job scheduler. It will now monitor all of the inference infrastructure, hand out jobs as needed and retrieve the results. We created a little fun visualiser for ourselves to see what STYX sees as it hands out work which we thought would be interesting to show below.

So what is the net benefit of all this work? well the main thing is we can once again fully examine every single address we receive from customers within our post-processing inference engine and we can easily add more servers to the inference infrastructure as needed in the future which is something we will need to do as the service becomes ever more popular.

One of the quickest ways to see the results of our new infrastructure is to check out the threats page. This is where we post only addresses our post-processing inference engine found to be proxies and it takes a random assortment of the most recent few hundred to be displayed there. It wasn't so long ago that all the entries on that page would show as last being seen 8 to 12 hours ago but with the new engine steaming through data we're discovering more proxies per hour than we used to discover per day.

This is why you'll see a lot of addresses on there were last seen just an hour ago or less. Being able to obtain knowledge of proxies like this that are "undiscovered" on the wider web (ones we've discovered that aren't yet posted publicly on message boards, blogs and websites) is important to us as it's these proxies that are perhaps the most dangerous and most likely are being abused by the individual[s] who set them up in the first place (often on hacked remote servers and IoT devices).

In addition to broadening our infrastructure we did also rewrite the way we synchronise information within our cluster. We found with so much data being updated per second there were some bottlenecks which we were able to completely solve several days ago.

Some of this was caused by the immense data changes occurring due to the new infrastructures ability to process so much data at once and some of it was a watershed moment caused by some internet problems affecting one of our cluster nodes that meant it had more data to synchronise than usual once it came back online, during this process we noticed how it wasn't able to reach parity with the other nodes after several hours due to just how much data was changing during the synchronising process.

So that's what we wanted to share with you today, bigger and better infrastructure that leads to tangible improvements in proxy detection.

Thanks for reading and have a great week!