Ziff Davis Internet
News & Resources for the IT Reseller
NewsReviewsTech AnalysisCommentarySecurityLinux/Unix
My Account |  

Misplaced Caution About Open-Source Anti-Spam?
By Fred J. Aun

Given that the new open-source approaches to spam filtering are capable of virtually eliminating unwanted e-mail and preserving the good stuff, why do many companies continue to struggle with spam?

Jonathan Zdziarski, the developer of the DSPAM open-source Bayesian spam blocker, believes IT departments of most small- to medium-sized businesses are afraid to try free programs or meet resistance from higher-level company executives.


"Most mid-sized companies just pull an appliance off the shelf," he said, noting there are "a million anti-spam companies out there with boxes loaded with a hodgepodge" of solutions. "That's one of the reasons these businesses, if you ask them, are convinced spam filtering is ineffective … A lot of these companies are running technology that's five to seven years old."

Some popular, commercially-distributed solutions say they employ Bayesian filters. When used alone, as in DSPAM and other similar program, these filters use statistical analysis to yield incredibly accurate spam control.

The Channel Insider Special: Managed Services in the Channel

However, Zdziarski, the author of a new book titled "Ending Spam," asserts most of the time the Bayesian filtering in these "hybrid" commercial products, if present at all, is rendered virtually ineffective because it filters only the mail that finds its way through the commercial programs' outdated "heuristic" filtering layer.

Good mail, or "ham," that was improperly deemed to be suspicious by the heuristic filter may never reach the Bayesian filter layer. This prevents the filter from learning what makes good mail "hammy" and it further increases the application's error rate.

The ability to tell good mail from spam is one of the most touted attributes of open-source spam-blocking programs using Bayesian statistical filtering as suggested by Paul Graham in "A Plan for Spam." Anyone who's ever absent-mindedly deleted an important e-mail that was improperly routed to a spam bucket can relate.

"False positives are innocent e-mails that get mistakenly identified as spams," wrote Graham in his paper three years ago. "For most users, missing legitimate e-mail is an order of magnitude worse than receiving spam, so a filter that yields false positives is like an acne cure that carries a risk of death to the patient."

The Channel Insider: Solution Builder

Zdziarski contends the Bayesian element mentioned on the boxes and ads for most commercial spam blockers "is more of a marketing term, really, than any type of real component of the solution." That's because a company selling an adaptive Bayesian spam filter would have a tough time staying in business.

"A true, adaptive solution is like fine wine," Zdziarski said. "You can take a tool like DSPAM … install it in a system, stick it in a closet and let it do its job with just basic user input. Let it sit there and run, without upgrades, for a couple years. When you take a look at it again, it will be performing better than it did on Day One."

Systems administrators and others facing a decision about e-mail filtering must weigh the cost of using commercial products against the fact that statistical language classification filters, while free, work best if users are trained to help them out.

Employees need to cooperate by "teaching" the programs the difference between spam and ham, a simple task that gets easier over time as the programs gain knowledge.

The Channel Insider Special: Building Solutions for SMBs

IT people must also determine the company's tolerance level for spam. Maybe 95 percent accuracy is good enough, even though it means up to five errors per hundred e-mails or 10 times more than would pass through a good statistical filter.

Zdziarski says he's sometimes unnerved by his filter's uncanny accuracy. For most systems administrators, the thought of employees opening spam containing viruses is something more scary, almost as bad as accidentally deleting that important e-mail from the CEO.

Tips-ITClick here for detailed comparison of anti-spam products (requires a fee to access)



Help boost your sales with BONUS PACKS.
Let BONUS PACKS help you sell more PCs—with up to $1,200 in valuable products and services. Visit
Buy Local Partner site >>

Join the Microsoft Empower program for ISVs.
A relationship with Microsoft allows you to compete with the big dogs. Learn more about resources, tools and training. Get the documentation >>

Test your licensing I.Q. and get a smart offer.
Take Microsoft's brief Windows® Licensing Pop Quiz and receive a free Windows Licensing Kit - only five questions. Take the quiz now! >>
Free Hands-On Training Lab
Find out how key features of SBS 2003 can help you open up a new line of revenue. Register now >>

SBS 2003 Sales Reference Card
This handy reference card contains features at a glance, sales objection handling, pricing guidelines & more. Get it now >>

Microsoft Empower for ISVs rewards your big idea with big benefits and support.
Access key development tools at a low cost to help you develop that idea into an innovative application. Learn more >>

Changing Business for the Better: A Practical Guide to BPM

This paper provides an overview of the benefits of BPM technologies and identifies the characteristics of BPM solutions that lead to successful BPM process-centric integration projects.

Download this free white paper to learn more!

>> brought to you by IBM

Attention Microsoft Solution Providers!

Want to gain a competitive edge? Try Microsoft Watch – FREE!

Each week you receive:
  • Microsoft News and Insider Information
  • Expert Analysis
  • Code Names of Upcoming MS Products
  • Year-Ahead Calendar, updated monthly

    Click Here to sign up now for your FREE 14 Day Trial to Microsoft Watch.

    White Boxes
    MS vs. IBM
    Linux in the Channel
    Stay in the Zone
    Put The Channel Insider on your desktop.
    Subscribe to The Channel Insider: Channel News, Reviews, Resources and more.

    Make your selections below:

    Contract Watch

    The Channel Insider Update

    Preferred e-mail format:

    Enter your e-mail:

    view all newsletters >>
    Channel Insider Quick Links
    Ziff Davis Footer Logo