Building a better Detective

Image description

One of the challenges a service like ours faces is the existence of anonymising services that specifically go out of their way to obscure their infrastructure. And so while it's easy to detect most addresses and their suppliers there's always a small percentage that slip by.

Which is why this past month we created a list of these difficult to index suppliers and went about building tools that were tailored specifically to scan and verify the addresses they offer. Traditionally when we want to scan a provider they will offer a webpage of addresses or hostnames which makes it easy to scan them and correlate what we find across our honeypot network and other scraped websites, this is part of our collect and verify strategy.

But some of these let’s say hardened providers will either mask their addresses behind signup pages, paid memberships or other means. For instance it's becoming very common for VPN providers to only show you their server addresses once you've signed up and paid for service and with there being hundreds of VPN suppliers paying for all those subscriptions isn't really commercially viable.

But even the free providers are becoming more shrewd by inserting randomly generated addresses within their legitimate address pools to thwart page scraping and some sites only show you addresses once you verify you're not a bot by solving a captcha or require a javascript engine to decode the addresses before they're rendered on the webpage.

All of these are things we worked to solve this month with what we're calling our Detective. It's a new module within our custom scraping engine which allows for a lot more thought during collection and processing. The results have been quite promising with our list of detected proxies and virtual private networks steadily increasing since it went live.

Some of its features include:

  1. Web and non-Web collection for anonymising services that only offer an application for accessing their network of servers.
  2. Javascript engine for solving any kind of proof-of-browser anti-Bot measures during address collection.
  3. Captcha solving support using image recognition with a fallback to human based solving.
  4. Bad/Fake/Generated address discardment through time based observation and frequency of appearance.
  5. Pattern recognition for indexing VPN providers infrastructure based on a few hand entered sample hostnames.

There are some very well known providers that are constantly being abused that have employed one or more of the above tactics to make it difficult for services like our own to get a full picture of their infrastructure but the new system we've devised has been able to break all of these approaches.

As always if you've come across an address, range or service provider we don't yet detect please contact us, we really do investigate every lead sent to us by customers.

Thanks for reading and have a great week!

API Version Selector added to the Dashboard

Image description

After we updated our API yesterday it became apparent that some customers had really depended on our type responses only being present for positive detections which is understandable because we hadn't used this field for clean addresses before.

Because of this we've decided to push up our launch of the API version selector which we mentioned in yesterdays post to today. This means you can now choose which version of our API you want to be used (by the dates of major revisions) and when you select a version you'll get a neat explanation about what changed compared to the previous dated version like in the screenshot below.

Image description

We are also providing a way for you to select which version gets used by adding &ver=date to the end of your queries. For example &ver=17-June-2020 and when you do that you'll get a version response back from the API like so:

{
    "version": "17-June-2020",
    "status": "ok",
    "98.75.2.4": {
        "proxy": "no"
    }
}

Which lets you know the version you requested was in-fact the version you received. This version indicator will not be present if you're using the latest version of the API via the selection box in the dashboard (the default selection) or you've not provided the &ver= flag with your query.

We hope this will help customers to plan for future changes to the API so they can upgrade their implementations when they're able to do so instead of on our release schedule.

Thanks for reading and have a great week!

API updated with new type responses

Image description

Today we've launched an update to our v2 API which includes new type responses including Residential, Wireless, Hosting and Business. These new type responses are to help you create better custom rules that target specific connection types beyond just the ones we've determined to be proxies or virtual private networks.

We'd like to focus just on one of those new types for a moment, it has been a long requested feature that we identify wireless connections because many customers have problems with malicious users who utilise wireless access points to get around IP bans. This has been because it's very easy to acquire a wireless connection and it's even easier to get a randomised IP Address from a cellular network provider.

And so we've added that type response alongside residential, hosting and business. Like all our types you shouldn't expect a type response to always be there as we won't show you any kind of generic or default response if we don't actually know what kind of a connection an IP is utilising.

In addition to this change we've also made a lot of back-end changes to how the API functions, specifically around storing and retrieving data (which includes our new type information even for clean addresses and location data) and also how the API is initiated on our webserver including API sub-versioning which isn't yet exposed to customers but we are using it internally and it is our intention to make it available to customers in some manner in the future.

So as of right now the new types are live which means you can build custom rules within your dashboard to utilise them immediatly. We've already updated our API documentation ahead of todays launch as-well including the test console where you can try it out in your web browser.

Thanks for reading and happy querying!

New design language & forthcoming API improvements!

Image description

If you visit the homepage today you may notice it looks a lot different to the one we've been serving for almost two years now and that's because we felt it was time to iterate on our design language and follow some of the newer things we've been doing particularly in the customer dashboard.

We want things to not just look nice but be consistent so that all of our pages and the features found on them feel part of the same product. We decided to start with the homepage as it may be considered the most important page of a website when viewed by potential customers.

So what have we changed? Firstly we've changed our call-to-action section which draws your eye to use a wider but softer drop shadow and increased the radius on border edges. Both of these changes give the homepage a more pleasing and modern appearance.

In addition to that we've done away with needless single-pixel borders around some of the information we showed instead embracing white-space and using soft pastel backgrounds beneath important information such as our API URL and our Live API Result area.

But it isn't only the aesthetics we've improved as the live api area now includes buttons for some example addresses and the API URL now updates in real-time when you change the address you're checking. We feel both changes will help to make a much better first impression.

As we mentioned this is simply the first page of the site that is receiving our updated design language and it will be implemented on other pages soon too so make sure to check back for that!

Image description

The other thing we wanted to talk about is a forthcoming improvement to the API, a huge change in-fact. Later this month we'll be enabling a new range of type responses for clean addresses so you can determine if an IP Address belongs to a residential address, business, wireless operator or hosting provider.

These new type responses have been on our roadmap for a very long time and it has taken considerable effort to provide this data with a high enough confidence level that it's not just usable but reliable. We first began writing code to implement this feature a little under a year ago and it has taken until now to reach a point where we feel it's ready.

But it's not just about the data itself as whilst we prepared to deliver this feature we went through the API and overhauled some of the ways we store and access metadata about all addresses (ISP, Location, Type, Threat etc). This was necessary to make available the new connection type information for all addresses, not just the bad ones. The resulting improved code will help us to deliver new kinds of metadata and in a more timely fashion in the future which is something customers are always asking us for.

The new clean type responses once available through the updated v2 API will be accessible like all our features to all customers whether you're on a free or paid plan. That means we'll be one of the only API's in the world offering location, provider, connection and anonymity information about IP Addresses for free. This is something we're unequivocal about, free matters and we are committed to our full-featured free offering.

That's everything we wanted to share with you today, please check back soon for another blog post where the new clean type feature will be going live. Thanks for reading and have a great week!

Improving Account Security

Image description

Last month we made a post where we told you about a new feature within the dashboard that rewards you with extra custom rules when you secure your account with a password and a two-factor authenticator.

Today we're bolstering your account security in two major ways to help combat account takeovers which have been steadily rising over the past year.

The first change is if you have a two-factor authenticator attached to your account then you're no longer able to create an account recovery code through our automated process here.

Instead when using the account recovery page you'll be sent an email where a support representative from proxycheck.io will accept evidence of account ownership from you and the recovery request will be manually evaluated. This is to stop situations where someone compromises your email account and then has the ability to gain access to your proxycheck account through that chain of access.

The second change we've implemented is login security alerts. From now on when you login to the dashboard using an address we've not seen you use before we will send you an email detailing that login so you can quickly take action if it wasn't performed by you.

As we mentioned at the start of this post account takeovers are on the rise. As our service becomes more popular so do the attempts on your accounts. We've seen a large increase in credential stuffing and so it's very important that you secure your accounts, we really cannot stress that enough.

Thanks for reading and we hope everyone is having a great week.

Switching email providers from Mailgun to Amazon SES

Image description

Today we've made a major change to the way we send email by moving from Mailgun to Amazon SES (Simple Email Service) and we wanted to explain why we've made this change and discuss some of the challenges we've faced with Mailgun as we feel sharing our experience may be beneficial to others.

Firstly though we should explain why we use a third party email service to begin with instead of self-hosting and delivering our own emails directly. Put simply it's to guarantee our emails actually get delivered. That isn't to say we couldn't send our own emails from our own server infrastructure as we certainly could but with the way so many email services are setup the deliverability relies on them trusting your reputation as an email sender.

There are many factors that go into an email servers reputation and one of the biggest factors is the IP Address it sends emails from. If you're sending email from an IP Address which is shared with spammers your emails are more likely to get bounced or blocked by the recipients mail provider. And some spam-blocking services that these mail providers use don't only list singular addresses but entire subnets, zones and even service providers.

This can create a headache for developers like us when we just want to make sure our legitimate email that you expect to receive from us actually arrives and so services like Mailgun, Sendgrid and Amazon SES are paramount to making sure the emails we need to send actually arrive in your inbox.

So why the move from Mailgun? - Well over the past year we've had many occasions where Mailgun's shared address pool has been blacklisted by major email providers including Google Gmail and Microsoft Outlook. Each time we have had to manually intervene raising tickets at Mailgun to receive a different IP Address. Often their responses have been extremely delayed and without much care to the situation or its importance.

On top of that their business model recently changed and their pricing per 1,000 emails sent is now 10x that of Amazon SES while offering an inferior product. In-fact just to get a dedicated IP Address at Mailgun it now costs $80 USD per month while at Amazon the same is only $25 USD per month.

For us the pricing isn't the issue though, we understand the value of a premium product but we felt that Mailgun failed us on too many occasions and their lack of care and attention to their own business negatively affected our own and caused a number of our customers to email us in dismay that they missed query overage notices, late payment notices or emailed our support questions and did not receive any replies from us.

They made us look bad to our own customers because they couldn't do the one job their entire business is setup to provide. This isn't meant to be a review of the Mailgun service but if you're reading this and considering them we would highly recommend going with one of their competitors.

So from today all email we send will go through the Amazon SES infrastructure and we are confident our deliverability rates will be high once again. In addition to this we would like to sincerely apologise to all our customers who were affected by the email problems we've experienced over the past year and in particular over the past two weeks which have been markedly bad and the reason we've spent the extra engineering time on changing mail providers.

Thanks for reading and have a great day!

Incentivising Security and updated Terms of Service

Image description

Here at proxycheck.io we take account security very seriously which is why we only allow you to login to your account using your email address once you have protected your account with a password and why we offer strong two-factor authentication. It's also why we always email you when any big changes are done to your account such as changing your email address, password or API Key.

We do all this because we know that when you implement our service into your own if someone were to gain access to your proxycheck account they could wreak havoc with your services availability.

And yet when we look at the percentage of customers who have protected their accounts with a password we were surprised to see that only 35.09% have chosen to do so. With that being just over one third we think we can do better and so today we've launched a new feature to the customer dashboard which gives you an extra custom rule for securing your account with a password and another custom rule on top of that if you've gone the extra mile to attach a two-factor authenticator to your account.

Image description

For those of you on our free plans this means you'll go from 3 rules to 5 which is quite an increase for free! - These extra custom rules are accessible to all customers including those who already have custom rule plans, they'll simply be added on top of what you already enjoy.


The second thing we wanted to discuss is our Terms of Service. They have been updated and none of the terms have actually changed but we've expanded on the terms with detailed descriptions which brings the page more in line with the look of our privacy policy and GDPR compliance pages including our change log feature so you can stay informed about changes to the page.

June Newsletter

Image description

Today we sent out the first newsletter of this year to users who have the "New features and improvements" email toggle enabled within their dashboard. This has again been the widest distributed newsletter so far with 50% more customers toggling the option on within their Dashboards than for our previous November 2019 newsletter.

If you didn't receive the newsletter but would like to read it you can do so here on our website.

We've made quite a few changes since November 2019 when we sent our last newsletter. We only publish two per year so you can expect our next one around December this year.

Thanks for reading and have a great week!


Back