Before we get into the new infrastructure, we just wanted to recap what has happened to our prior Asian infrastructure and why we're making a change.

Just over two years ago, we introduced two servers into our infrastructure to serve our customers in the Asian and Oceanic regions. We rented servers in Southern Asia, specifically in Mumbai, India, and they served us moderately well during that period. Although both servers experienced storage device failures early on, we persevered with them.

However, starting in May this year, we began to experience significant network degradation with these servers, which was noticed by customers and caused significant delays in API response times, in addition to failed requests. Our network partner CloudFlare, which acts as our Content Delivery Network (CDN) were made aware of the network faults and successfully intervened to fix them.

However, the entire saga lasted more than a month, and in fact, we brought online an emergency server at a different datacenter in Singapore at significant cost in a bid to reduce latency for customers while the network issues with our Indian based servers were ongoing.

Fast forward to early June, and the network issues were resolved; we switched back to our Indian servers and continued to operate like normal. But then, a few days ago, the network problems came back, and so we once again rented a temporary server in Singapore to handle traffic while we waited for a response from our Datacenter and CloudFlare.

Ultimately, we feel that these issues with our servers in India have reached a threshold where we can no longer tolerate their problems, and, looking at our traffic over the past 60 days, we can see that even when they were working properly, there were still spikes of intermittent connectivity and high latency. So we've decided to get rid of them and go all-in on Singapore.

This decision wasn't taken lightly, as Singapore is quite literally 8x more expensive for both hardware and data than India was for us due to various incentives we were provided to utilise servers based in India by our datacenter host. But ultimately, it doesn't matter how good the pricing is if the servers can't be used and are providing a bad experience to customers.

In addition to performance as a reason, we looked at our Asian traffic mix and saw that 85% of all our Asian traffic was, in fact, from Singapore. It's a popular destination to host servers, and obviously, a lot of our customers are there. It's also physically closer to Australia and New Zealand, which were also being served by our Indian servers.

So not only will the network health be better, but the latency will be greatly improved for our Asian and Oceanic customers just due to geography. Now with this move, we are going from two servers to a single server, but our load balancer will failover to our European infrastructure if there were to be an incident, so we're not compromising on uptime or availability.

Since we're going all in on Singapore, we decided to rent a very performant bare metal server and not use a VPS. This way we get uncompromising performance, and it's a like-for-like replacement for the previous Indian servers, which is to say this one server is faster than both Indian servers combined, and it's, of course, powered by AMD, like most of our servers have been over the past few years.

The new server was deployed today, which you can view on our recently revamped status page. Our focus right now with infrastructure is to purchase high-powered servers that can handle lots of traffic and requests instead of lots of low-powered servers that may struggle individually. This also suits our performance requirements, where we're always looking to get the request latency of the API down as low as possible.

So that's the update for today. I also just wanted to reiterate what was alluded to in the last blog post, we do have a significantly big announcement coming in the near future, we're hard at work on the next chapter of the service, so check back next month for your first look at that.

Thanks for reading and have a wonderful weekend.

Over the past several weeks we've been adjusting the website's appearance and usability. Today, we'd like to share some more changes we've made since our last blog post.

Signed In Devices

A few years ago we added a feature to the bottom of the customer dashboard that allowed you to sign out any devices that were signed into your account. Essentially erasing their sessions server-side so their access to your account can be revoked even if you don't have access to those devices anymore.

We wanted to expand on this feature by actually showing you the devices which are signed into your account, what browser and operating system they're using, the country and region they accessed your account from and the last time they accessed it. To that end, we've added a little blue question mark next to the "Sign out everywhere" button which will show a little pop-up like in the screenshots below illustrating the same view on our dark and light site themes.

We decided to make this feature ourselves in-house instead of relying on an external service or product. This means we will maintain the list of device characteristics ourselves. To begin with, we're supporting all the major browsers (Firefox, Chrome, Edge, Safari, Opera, Brave, Internet Explorer) and also the major operating systems (macOS, Windows, Linux, Android, iOS, iPadOS, ChromeOS etc). But we've also added some specific detection for particular devices so for example if you're using an iPhone it will just say iPhone instead of iOS. We've also done this for iPad and some other product categories.

This device data is stored server-side with your normal account information and when you clear your signed-in sessions (either one at a time or everything all at once) the data associated with the erased session (browser, os, location, last seen time) will also be erased from our servers at the same time. We're doing it this way because this feature isn't intended for us to track you but for you to track access to your own account.

Confirmation Dialogs

We've had confirmation dialogues for a few important areas of the Dashboard including changing your API key, upgrading or downgrading your paid plan, cancelling a paid plan and erasing your account. We did this to make sure unintended changes weren't made by accident. But customers have told us they would like these confirmation dialogues in more areas specifically when they try to erase a Custom Rule or Custom List.

We completely agree and so we've added confirmations to both of those features, when you try to erase a rule or list a modal will appear with a darkened background which asks you to confirm or cancel the removal like in the screenshot below.

Custom Rules API

A feature we've been asked for a few times since we introduced Custom Rules has been a way to interact with your Custom Rules via a Dashboard API like you can with Custom Lists and CORS domains. The reason we've not added this feature is because Custom Rules are very complicated and we feel the user interface within the Dashboard is the best way to create and alter Custom Rules.

However, there are things we can make accessible that do make sense to enable through an API. Viewing which rules you have in your account and whether they're enabled or not, exporting rules the same way you can through the Dashboard interface (for manual editing or later re-import) and toggling rules on or off.

To support these use cases we've added a new Dashboard API endpoint which you can view all the documentation for here. We've also taken the time to update our official PHP library to support this new Custom Rule Dashboard API. You can obtain that from packagist here or by using the relevant composer command within your environment.

So that is all the updates for today, we do have some much bigger things being worked on alongside these more minor interface changes but it will likely be a few months before we're ready to share those with you. Until then we hope you liked todays changes and have a wonderful rest of your week.

Over the past week, you may have noticed the site receiving some design updates. The most significant of these is the new top navigation bar, where we've made many usability improvements.

The New Navigation Bar

Firstly, we wanted to keep the colors the same. We consider the red-to-blue gradient part of our brand, and so while we did experiment with some monochromatic navigational bars with blurred transparency, we decided to stick with the strong colors we've always had. Of course, it still supports both our light and dark site themes.

Secondly, we needed to declutter the navigation bar which meant moving some of the lesser accessed links to dropdowns. We didn't want to just have basic list dropdowns though so we created cards with miniature screenshots of the pages with matching header colors and descriptions. We still kept the most important links always accessible though including the Dashboard which is accessed most frequently by customers.

Thirdly, improving the mobile experience. The "hamburger" menu as it's commonly referred to was difficult to use on mobile devices due to how we had implemented it. With this total rewrite of the navigation bar, we were able to significantly improve the menus reliability on mobile and make it look much nicer too. We also had enough room to keep a dedicated Dashboard button present while on mobile.

Fourth, legibility. The previous navigation bar used thin text that was difficult to read against our bright background gradient. We have now switched to a more bolder font style and we've introduced our staggered dropshadow present on other page elements to the text of the navigation bar aswell. This makes it much easier to read and we've also done away with reducing the size of the font if your display size is smaller than average, we instead reduce the whitespace between menubar items before finally stepping down to the mobile interface. This keeps the navigation bar text legible on all screen sizes at all times.

Fifth and finally, site-wide address lookups. This has been something we've wanted to add to the navigation bar for a while, the ability to lookup an address from any page on the site. We've renamed our threat pages to lookup pages as this much better represents what these pages are for.

Design is an iterative process and we will likely perform a few more tweaks to the navigation bar over the coming months.

Status Page Updates

Over the past several weeks we have been dealing with network issues with servers in our Asian service region and due to that, we had to take the entire region offline and forward traffic to our European servers. This resulted in our customers based in Asia facing higher than acceptable latency and although our status page indicated our Asian servers were offline for maintenance there wasn't any reason provided for why that was the case.

To remedy that we updated our status page with a new incident messaging feature that lists current incidents at the very top of the page and past incidents at the very bottom providing both current and historical information. We spent some time on the presentation, colors and iconography of this feature to make it as accessible and understandable as possible. Our key goal was to provide you with the exact information you need in as concise a way as possible.

So those are the updates for today we're constantly re-evaluating the service and that includes design, making sure we're staying current with modern expectations and maintaining a high level of usability and efficiency in our designs, we think we've struck the right balance with these updates and hope you think so too.

Thanks for reading and have a wonderful week!

Today we've pushed live an update to the API that improves performance and we wanted to talk about it. Over the past two months we've been evaluating every aspect of the service to locate performance bottlenecks and correct them. You may have seen this blog post we made on April 10th about this where we changed a lot of things related to storing customer data and the customer dashboard.

That work has been ongoing, and we've been squashing bugs and handling edge cases that resulted from those changes. We've also been rewriting significant parts of our data synchronisation system, and as a result, we've been able to significantly lower CPU usage on all our servers. Essentially, our servers now spend less CPU time on database stuff so they can put more resources into answering your API queries.

However, there was one area we had not changed in a while. In this blog post from November 2022 we mentioned how we were preparing to deploy PHP v8.2. We did deploy PHP v8.2 on both the API and our website the following month. However, we've stuck with PHP v8.2.x versions since then. Meanwhile PHP v8.3 and now v8.4 have been released and we were lagging behind. To be clear we still used a supported version of PHP that has been receiving security patches and bug fixes, but we weren't using the latest major versions that contain improvements to the language, new features and crucially for us, extra performance.

Today however that all changes as we've upgraded all the way to PHP v8.4.7 the very current release and in doing so we've improved API response times. Previously for our servers that achieved an average of 5.7ms internal processing time (network overhead not included), we've been able to lower that to an average of 4ms. While this isn't a lot in time terms as a percentage that's a 29.8% reduction in average compute time.

In addition to upgrading to PHP v8.4.7 across the entire site (our Dashboard and other pages will also benefit from these speedups and in fact, should benefit even more than the API due to their larger footprints) we've also been optimising the API itself. We've eliminated two disk read operations which checked for the existence of data that no customers made use of, these reads had nanosecond level impacts but extrapolated across all the queries we handle it adds up.

Similarly to the above optimisation, where we couldn't eliminate a disk read we instead moved the data to our in-memory data store system. Accessing data from memory is much faster than from storage disks (even though we exclusively use NVMe SSDs) and it allows for higher throughput of the API by not needing to wait on storage queues. In fact, our in-memory data store is fully multithreaded and non-blocking too. We were able to offload three separate database reads from disk to memory with this change and at least two of these are used by every single query to the API.

We've done extensive testing to make sure these changes especially the move to PHP v8.4.7 are fully tested. We performed millions of varied queries against the API and tested all our web pages and most of the functions within those pages before pushing it live. As always though it's possible we missed something so if you notice anything not working correctly please contact us, we do respond to every communication we receive.

So that's the update for today, lots of performance improvements and a major server-side code interpreter update, thanks for reading and have a wonderful week!

Today, we have introduced two new high-end servers into our North American service cluster called Vega and Nova. These servers are based on AMD's EPYC Genoa platform like our previously introduced Rigel system a few months ago.

By adding these new servers, we're increasing our North American capacity by 50%. And similarly to our previous rationale for adding higher end servers we wanted more buffer against burst usage by customers who are under momentary attacks.

These new servers share the same host as our Rigel server we introduced in February but they don't use the same host as our three other North American nodes (Lunar, Saturn and Jupiter) which helps to insulate us against any single host having unforseen network or power issues.

So that's the update for today, thanks for reading and have a wonderful weekend.

Today we wanted to talk about some changes we've been making behind the scenes to help us scale to meet the needs of our growing customer base. You may have noticed over the past week that the customer dashboard has not been working as reliably as it should with changes made within your account not synchronising amongst all our infrastructure in a timely manner.

We've also had issues with new users signing up only for their accounts to not be activated by our internal systems, resulting in them being unable to log in and use their accounts. Both of these issues are linked to the same problem, which has been the unreliable synchronisation of data amongst our nodes for our most high-traffic database operations.

This includes things like signing up, changing any setting in your account and the creation or modification of user-generated content (custom lists, custom rules, CORS domain entries etc). We put these things in a fast-track lane so that the Dashboard always feels snappy even when your dashboard requests could change from one server to another in our cluster during a single user session.

But as we've grown we are seeing more and more traffic targeting the dashboard and this once super-fast lane began to slow to a crawl culminating in this week where changes and additions to customer data were greater than the ability for our servers to synchronise those changes in real-time.

To solve this problem we looked at the way we've been handling requests to and from the database and identified many pain points. The biggest one was the amount of database transactions for a single dashboard web request this includes both loading the dashboard and also creating or modifying content within it and then sharing those changes with other server nodes.

By restructuring our customer data and coalescing the gathering and saving of that data into single database operations we've been able to reduce traffic between nodes by a factor of 7 on average for a user accessing their Dashboard and by a factor of 3 to 5 when they make changes depending on what those changes are.

As a result of these changes, the dashboard and user signups are now being handled in real-time again. In addition to these changes for our most accessed customer data, we've also been working on the slower database synchronisation we use for big data. This includes things like customer positive detection logs.

One thing we noticed here is a lot of this data is rarely accessed but the enormity of it was significantly delaying our ability to bring up new nodes (due to the need to synchronise all this data) and the ledger size of our database that maintains a listing of what is synchronised and which nodes are missing data was getting very large and becoming a burden for our server nodes to handle.

To solve this problem we've begun to compress all high-impact user data and as a result we've been able to reclaim 80% of the disk space utilised by this data. This has also had the side-effect of making this data much faster to access as even though we use high-end Solid State Drives on all our servers, storage is still the slowest component out of the CPU, RAM and Storage in a server. So by decreasing the size of the data we're loading from our disks, we can load it into memory much faster.

In addition to the compression, we've also altered how our database blocks work. Prior to this week, all data in the database were stored in 8MiB blocks. This made things mathematically simple. But as user data has grown the amount of blocks has increased to an unmanageable amount. Due to this, we've now moved to an adaptable block size between 8MiB and 64MiB with customer data being placed into appropriately sized blocks depending on their data volume. Since servers need an entire block before they can access the data inside we will adapt what size block is used for a specific customer's data based on their data volume with smaller blocks able to transition into larger ones as their data grows.

So this is the update for today, we're hoping that there won't be too many teething problems but to be honest with you the database scheme updates that apply to the Dashboard are rather major and were introduced faster than we would have liked due to the serious performance degradation we were seeing. What this means is, there may be bugs and we ask for your patience and diligence in reporting any you find.

Thanks for reading and have a great week!

Today, we have introduced a new high-end server into our European service cluster called Titan. It is named after Saturn's largest moon, which boasts a thick atmosphere and a landscape of liquid hydrocarbon lakes, making it quite unique in our solar system. This server is based on AMD's EPYC Milan-X platform and is currently the most powerful server in our cluster.

By adding this new server, we're increasing our European capacity by 20%. The main reason we've done this is that we wanted more buffer to withstand momentary increases in API usage, which sometimes happens when our customers come under targeted attacks.

This new server also further increases our provider diversity as it's not hosted by the same datacenter partners we've been using for any of our prior nodes and it's deployed in a different country to our other nodes as well (we are now using three different countries for our European nodes).

As we mentioned in the introduction of our Rigel node two months ago, we will also be refreshing a server later this month, that server will be our North American node called Lunar. We intend to keep the node name the same but it will be transitioning to a different provider and more capable hardware based on AMD's EPYC Genoa platform which Rigel already uses and we've been very happy with.

So that's the update for today, thanks for reading and have a wonderful weekend.

Today, we would like to share a big improvement we have made to our API's disposable mail-checking feature. We launched this feature a few years ago and it lets you check whether an email address belongs to a disposable or temporary mail service so that you can deny service to uncontactable users.

The biggest roadblock we've encountered since launching this feature has been the collection of disposable domain names from the hundreds of sites that offer temporary mail services. The reason for that is their increasing use of anti-bot technology, which made it hard for us to collect their domains in an automated fashion. Often requiring us to collect them by hand which meant there were large periods where certain temporary mail services were underrepresented in our data.

That however has changed over the past several days since we deployed a new gathering system which utilises AI agents to browse these sites, solve captchas and other anti-bot screens to collect the domains. More than half of the sites we profile (and there are a few hundred) employ some kind of anti-bot screening or captcha system.

But with our new AI agent-driven gatherer we've been able to add hundreds to thousands of new domains to our database every day since deployment thus making our disposable mail system much more effective.

So that's the update for today, we hope you'll try the disposable mail detection and feel free to let us know what you think or if you've found a disposable mail site that is giving you results we don't have, send us a message and let us know, we'll build a profile for them.

Today we've introduced a change to the API version selector found within customer dashboards. This change now clearly identifies which versions of our API are supported and which ones are no longer supported.

We've made this change for two reasons, firstly so that we can stop supporting older versions of the API which will free up more resources for us to put into newer versions of the API. And secondly so customers can have greater control over when to upgrade their API version to gain access to new features and upgrades.

To expand on that second point, previously we've only ever issued new versions of the API when we make a change to the JSON output format that we believe could break implementations by our customers. But now we will be issuing new API versions when we make any kind of substantial code change (anything beyond maintenance or bug fixes), even if it doesn't alter the output format.

The reason for us wanting to release more API versions like this is to insulate our customers from API mistakes and errors as we introduce both new features and changes.

And so by giving our customers greater control over when to upgrade for even minor new features and changes they can be protected against bugs that we may have missed during development and testing. We felt the need for this as we introduced our new location engine last month which did have some problems at release that required us to temporarily roll back the update and then implement rapid fixes before redeployment.

We're very fortunate to have understanding and patient customers who made us aware of these location data regressions and allowed us time to fix the problems. While reflecting on their feedback we realised that bundling the new location data update into our previous release (which introduced device estimates) was not the right thing to do and we should have instead issued a new major version of the API alongside this feature.

So above on the left is the new API version dropdown selector and on the right is what it looks like if you choose an unsupported version of the API. Below is what you'll see if you're already running on an unsupported version of the API.

We wanted to make sure these notices couldn't be easily missed thus the big red warnings. Our support timeline for API versions going forward will be a minimum of three years from the point of a release. Currently, we're supporting every version since May 2022 because that was when a large shift happened in the codebase with the introduction of Custom Lists. We will likely support versions beyond this date range depending on how time-intensive it is to port changes back.

So that's the update for today, as always thanks for reading and have a great week!

A couple of days ago, we introduced a new high-end server into our North American service cluster called Rigel. It is named after the blue supergiant star in the constellation of Orion. This new server is based on the AMD 4th generation EPYC Genoa platform, making it one of the fastest systems in our fleet as of today.

By adding this new server we're increasing our North American capacity by 33% as we've seen traffic there increase significantly over the past 12 months and we like to have a large buffer so we can take some servers offline for maintenance should it be needed.

This new server also further increases our provider diversity as we're utilising a new datacenter which is operated by a different provider to our other North American server nodes.

In addition to Rigel, we think around April we'll be refreshing several of the servers we picked up two years ago which will help raise overall performance, lower latency and give us more room to grow.

That's the update for today, we should have another non-infrastructure-related update regarding the service directly next week so stay tuned for that.

Thanks for reading and have a wonderful weekend.