In late 2009, eFolder quietly acquired the DoubleCheck Email Manager product, which is an email filtering and management service, hosted in the eFolder cloud, a partner’s own private cloud, or on an on-site hardware or virtual appliance. We’ve been keeping DoubleCheck under wraps since then on purpose, quietly enhancing every part of the product. This post lifts the covers on what happened to DoubleCheck in 2010. More later on what exciting things we have planned for 2011.
If you’re an eFolder partner reading this, you may not have even heard about DoubleCheck yet. Yes, we’ve been really quiet about it. We’ve been integrating it into the eFolder product family and making a lot of cool enhancements, to ensure that when we deliver, it will live up to the eFolder mantra of delivering services that intuitively just work the way you would expect it to, with very little ongoing maintenance required.
So why did eFolder acquire DoubleCheck? It fits well into our long-term plans for building a suite of data protection services. The technology platform itself is very scalable, modular, and efficient. And the product was built from the ground up to be focused on providing branded service through partners, so it fits extremely well into eFolder’s channel-only, fully branded strategy. While there are many email filtering services out there, you can really only count on one hand those vendors that are 100% channel-only (you know who they are). We’re applying the same recipe that has brought success to our data backup and business continuity services to DoubleCheck: services that just work, excellent support, and partner-friendly business practices. We’re aiming to make your life simpler by offering more services under one umbrella, saving you time and improving your margins.
Now on to the good stuff, here’s what’s kept the DoubleCheck development team super busy this past year:
An adaptive statistical spam classification layer, leveraging hierarchical orthogonal sparse bigrams.
My, isn’t that a mouthful! Such a phrase really deserves a blog post of its own, so look for part two of this post quite soon with the details.
To summarize, different domains and even different users within the same domain can have drastically different views of what is and is not spam. Additionally, certain companies may work in industries (e.g., medical) that may frequently receive legitimate email with keywords that could otherwise be considered spammy. This new technology provides the means to automatically adapt to the individual patterns of users, while also still benefiting from the collective pattern analysis of domains, groups of domains, and users across the entire system. Thus, if most users across the entire system consider a particular email as spam, when similar emails are received in the future, it will decide for most users that these emails are also spam, but this decision may be overridden for specific domains or individual users whose past behavior indicates they probably think otherwise. The details are quite interesting, so we’re going to post more about this.
Secure yet password-less end-user self-management of quarantine and reporting of false negatives.
DoubleCheck should be easy for end-users to use, without causing an administrative burden on IT staff and IT service providers. We view quarantined email and the ability to interact with DoubleCheck as an extension of a user’s mailbox. If a user has access to their mailbox, they should also have access to DoubleCheck (for just their mailbox). With this feature, an end-user self-authorizes their particular computer by clicking on a link sent to them in a one-time activation email. When traveling, a temporary authorization can be performed, allowing interaction from computers they don’t normally use. Administrators can also configure password authentication (integrated with active directory) if desired.
So what can an end-user do? Release messages from their own quarantine, report spams that weren’t properly identified (to help train the new adaptive classification layer), search their own mail logs, and view reports. Which features they can access are controlled by their IT service provider. End-users also interact with the system through quarantine report emails and automated reports.
Administrator-enhanced training (optional).
Administrators that want an extra degree of monitoring and control can optionally choose to have email that may be spam (but the system is not sure about) to be BCC’d to an administrator address. The administrator can train these messages as spam or as clean in bulk by forwarding the messages to specific trainer email addresses. Many end-users will not want to use this system, but it’s available for those power users that insist on keeping a close eye on email in the “grey area.”
Additional real-time global spam insight and analysis networks.
We tap into analysis networks that process billions of email every day, allowing us to react to new global spam patterns within seconds. Reputation information of different elements is based on a number of real-time data sources, including honeypots, proxypots, known botnets and zombies, open proxies and relays, domain age, domain registration characteristics, and patterns of user-reported spam from those participating in a global spam feedback system. DoubleCheck had partnerships with global networks prior to 2010; however, we’ve added to our network and formed faster data synchronization links, improving the quality of the data. Additional networks were selected based on our quantitative analysis to best broaden our coverage of new areas and new types of spammers.
Real-time classification of hyperlinks in emails.
Most unwanted email includes one or more hyperlinks, referring users to sites that sell contraband, host phishing attacks, or host malware to attack visitors. We leverage our partner global anti-spam and anti-malware networks to scan every hyperlink in every email and block those known to be undesirable. Partnering with the right networks gives us as much reach and visibility into global spam patterns as any of the larger anti-spam filtering vendors.
URL de-obfuscation to unmask the true website a spammer is linking to.
Spammers go to great lengths to hide their “payload” (links to websites, images, advertisement text) from automated classification systems yet still have them be visible to users reading the spam. One such technique is to use URL shortening services one or more times to make a link to a malicious site appear benign. For example, http://bit.ly/YZL will actually redirect a user to http://www.google.com/, but could just as easily have redirected a user to a malicious malware or phishing site. DoubleCheck automatically unshortens URLs from dozens of known shortening services. (We’re adding more all the time.) We then analyze the reputation of the final target URL. We’ve also identified additional ways that spammers abuse these services and block messages with such abusive links.
Auto-whitelisting.
DoubleCheck performs both inbound and outbound filtering of email. As such, it can detect who users communicate with on a regular basis and ensure that email from these people are always allowed through. However, it is carefully implemented so that spammers cannot easily spoof the system and exploit this feature. Even if a spammer spoofs the return address of someone a user regularly communicates with, this alone is not sufficient to activate the auto-whitelisting entry. That being said, our system is smart enough to still recognize email from legitimate senders even as their location changes or as they send email from different sources (their office and their phone). This provides the best of both worlds, ensuring users receive the mail they want, while keeping the spammers out.
Under the covers.
Lots of effort was spent by the team adding smartness throughout the system to further increase reliability and self-healing. Full text log searches got a lot faster (10x faster or more in some cases). And backend classification performance improved by up to 10x as well.
In conclusion.
Whew! Are you still reading? Lot’s of exciting things are planned for DoubleCheck in 2011. We’re gearing up the sales and marketing engines. If you’re an eFolder partner, ask your sales rep about how you can save money through our multiple services loyalty discount.
