Ziff Davis Internet
News & Resources for the IT Reseller
NewsReviewsTech AnalysisCommentarySecurityLinux/Unix
My Account |  

Slamming Spam for Free
By Fred J. Aun

Open-source anti-spam applications that use statistical analysis can pick out spam much more accurately than Bayesian filters, once they get enough training.

Jonathan Zdziarski knows how open-source spam-filtering programs work. He wrote one.

Even so, Zdziarski's creation DSPAM—and similar programs that are designed to get better at their job over time—sometimes gives its creator the willies. It usually happens when the application blocks spam messages that appear absolutely legitimate at first blush.


"It gets really accurate after the first 12 months," said Zdziarski, referring to DSPAM's built-in ability to "learn" how to filter mail. "As you train it more, and as it learns more on its own, it gets to this level of accuracy that's really spooky."

Could DSPAM and other spam-blocking programs using statistical language filtering, such as POPFile, Bogofilter and SpamProbe, be the long-awaited killer apps for a world weary of unsolicited Viagra ads or Nigerian money scams?

Zdziarski, the author of a new book titled "Ending Spam," believes these new-generation programs have finally tipped the scales away from spammers and in favor of beleaguered e-mail users.

"Spammers have been trying to evade statistical filters for a few years, to no avail," he writes in his book. "As spammers change their messages to evade the filters, the filters always seem to have an eerie way of detecting their new tricks—usually without the end user even noticing."

The Channel Insider Special Report: Making Security a Business They do that with statistical analysis not only of the content and subject lines of the e-mail, but by using a form of artificial intelligence to learn and adapt to their users' personal preferences.

Most current commercial spam filters use Bayesian analysis to filter spam. These programs are usually based on a "heuristic" model, similar to most anti-virus applications, in which programmers compile lists of "detection rules" and blacklists used by the software to compare ingredients of incoming mail with telltale factors found in known spam.

Based on a theorem developed by 18th century mathematician Thomas Bayes—and suggested in 2002 in a paper titled "A Plan for Spam" by author Paul Graham—Bayesian language classification filters break down e-mail into smaller components called "tokens," which are assigned numeric values based on their "spamminess."

Savvy spammers are continuously concocting ingenious ways of tricking these outdated programs. For end users of heuristic spam blockers, the only way to keep their in-boxes relatively free of junk mail is to repeatedly update the software by downloading the latest detection rules and blacklists.

This scenario benefits the companies selling heuristic filters (and that includes most of the popular, commercially available offerings) because they require users to purchase subscriptions in order to download the newest detection rules.

"There are some applications out there that do a good job, but for the most part they are driven by this business model," complains Zdziarski. "Any [spam blocker] company out there that wants to stay alive needs monthly or annually recurring revenue."

The Channel Insider Special: Managed Services in the Channel

If a heuristic spam blocker is not injected with updates, it quickly loses effectiveness.

The programs also use historical data sets, which serve as their memory of "characteristics that the filter has learned to be identifying characteristics of spam [and nonspam]," Zdziarski writes.

The Bayesian filter's analysis engine is its reasoning tool, using input from the tokenizer and historical data set to determine whether messages are spam or ham (legitimate mail).

Zdziarski said most commercial spam blockers are about 95 percent accurate. A well-tuned statistical filter can consistently be 10 times more accurate. "It's a huge difference," he said.

Tips-ITClick here for detailed comparison of anti-spam products (requires a fee to access)



Help boost your sales with BONUS PACKS.
Let BONUS PACKS help you sell more PCs—with up to $1,200 in valuable products and services. Visit
Buy Local Partner site >>

Join the Microsoft Empower program for ISVs.
A relationship with Microsoft allows you to compete with the big dogs. Learn more about resources, tools and training. Get the documentation >>

Test your licensing I.Q. and get a smart offer.
Take Microsoft's brief Windows® Licensing Pop Quiz and receive a free Windows Licensing Kit - only five questions. Take the quiz now! >>
Free Hands-On Training Lab
Find out how key features of SBS 2003 can help you open up a new line of revenue. Register now >>

SBS 2003 Sales Reference Card
This handy reference card contains features at a glance, sales objection handling, pricing guidelines & more. Get it now >>

Microsoft Empower for ISVs rewards your big idea with big benefits and support.
Access key development tools at a low cost to help you develop that idea into an innovative application. Learn more >>

Changing Business for the Better: A Practical Guide to BPM

This paper provides an overview of the benefits of BPM technologies and identifies the characteristics of BPM solutions that lead to successful BPM process-centric integration projects.

Download this free white paper to learn more!

>> brought to you by IBM

Attention Microsoft Solution Providers!

Want to gain a competitive edge? Try Microsoft Watch – FREE!

Each week you receive:
  • Microsoft News and Insider Information
  • Expert Analysis
  • Code Names of Upcoming MS Products
  • Year-Ahead Calendar, updated monthly

    Click Here to sign up now for your FREE 14 Day Trial to Microsoft Watch.

    White Boxes
    MS vs. IBM
    Linux in the Channel
    Stay in the Zone
    Put The Channel Insider on your desktop.
    Subscribe to The Channel Insider: Channel News, Reviews, Resources and more.

    Make your selections below:

    Contract Watch

    The Channel Insider Update

    Preferred e-mail format:

    Enter your e-mail:

    view all newsletters >>
    Channel Insider Quick Links
    Ziff Davis Footer Logo