Bayesian Spam Filters Explained

Bayesian Spam Filters Explained

by: Niall Roche

In a word Bayesian spam filters are กintelligentก. Bayesian spam filters are intelligent in so far as they’re capable of comparing two sets of information and acting on the result. This is in direct contrast to the vast majority of other spam filters who use prebuilt rules to decide which email is spam and which is not.

Bayesian spam filters can take one group of legitimate email and another group of spam and compare the values and data of each. The definition of legitimate email that it creates at the end of this comparison session is what it uses going forward to scan your inbox for spam.

FYI Bayesian spam filters are named after Thomas Bayes an 18 century cleric who created something known as Bayes Theorem. In summary Bayes Theorem is as follows: ..กin statistical inference to update estimates of the probability that different hypotheses are true, based on observations and a knowledge of how likely those observations are, given each hypothesis.ก In plain English it looks for obvious repeating patterns to form an กopinionก on something. In spam filter terms that กopinionก becomes a rule which keeps you spam free (or pretty close 🙂

The really neat thing about Bayesian filters is that they’re capable of learning. For example if they decided to block an email because the filter perceived it as junk but the user marked it as valid mail the Bayesian filter then knows not to block that type of email in the future. So, in time, this type of spam filter learns enough to block spam far more effectively. AOL have embraced this type of spam filter with the launch of AOL 9.0 and AOL Communicator if the big dog wants it then it must be worthwhile?

So what Bayesian spam filtering options are available to you? Well quite a few to be honest and youกll be pleasantly surprised by some of the names involved 🙂 The first one on the list is AOL with their AOL Communicator product. The spam filtering features in AOL Communicator and AOL 9 are, to be honest, impressive. Think what you will of the provider themselves AOL Communicator is an excellent product and is suitable for use by both PC and Mac OSX users.

Next up we have Eudora. The nice folks at Qualcomm have designed an excellent email client that also has built in Bayesian spam filtering. Iกve used Eudora in the past and itกs a neat little package. Again the benefits here are advanced integrated spam filtering with your email automatically. Mac OSX and OS9 users are in luck with Eudora providing support for both.

If youกd like to know more about spam filters or just spam in general please drop by our site at www.spamsite.com

About The Author

Niall Roche is the content author and owner of http://www.spamsite.com.

[email protected]

This article was posted on December 21, 2003

by Niall Roche

Does Your Website Host Fight Spam?

Does Your Website Host Fight Spam?

by: Eric Lester

Virtually anyone with an email address knows what Spam is, and has, perhaps, considered giving up the speed, convenience, and simplicity of email because of it. Those who have their own websites are more vulnerable than the average person with a single work or home email address from their company or Internet Service Provider. Email addresses visible on a website can quickly become Spam magnets, as automated programs, similar in form to search engine spiders, roam the web, looking for addresses to which new broadsides of Spam may be fired. Website hosting companies generally provide their clients email accounts for use with their domain, but are you with a host that provides those email accounts with Spam and virus protection?

Local Blacklist Filters

Webmasters shouldn’t need to seek out local filters for their siteกs contact email addresses. There are a variety of server level solutions a hosting company can offer to protect their users from unwanted emails. A most basic step is provision of a very rudimentary กblacklistก functionality to their users, allowing them to prevent future Spam emails from arriving from the same address. This type of filter is virtually worthless in todayกs Spam environment, though, as it is quite rare to see unsophisticated email arriving from the same address multiple times. Spammers have grown far more sophisticated than that. Blacklisting functionality is only really useful in avoiding email from other real people you don’t particularly wish to hear from anymore.

Keywords and Regular Expressions

More advanced server level Spam filters are available. A small advance is accomplished using keyword filters. Keyword filters merely check for instances of a certain string of characters and deny the message if that string if found. The core problem with keywordonly filters is they can กover filterก. Someone who puts กsexก on their keyword filter will find receiving local news and event announcements difficult if they live in a town named กEssexก. Some filters attempt to address this deficiency by using ‘regular expressionsก in order to build a sophisticated rule set to prevent Spam from reaching your inbox. Briefly, regular expressions are syntax rules used to identify certain strings of text or numbers. These rules can be set up to identify text patterns that are commonly used in Spam. They can become quite complex, but, as with most any filtering method, are not 100% bullet proof. Some filters that use regular expressions come with a basic set that can be appended by the user. Obviously this kind of feature is of little use to someone not familiar with regular expressions.

Bayesian Filters

Currently the most sophisticated filtering methods use Bayesian inferences. Bayesian filters take a large data set and determine the probability a message is Spam based on its similarity to previous Spam messages. The more emails that are processed and flagged theoretically make the filter more accurate. Services that provide filtering on an ISP or host level, like Postiniกs กSpamAwayก, filter billions of emails and provide the highest level of success and fewest กfalse positivesก. SpamAway is already highly intelligent about identifying Spam and doesn’t require any กlearningก commands or examples be provided. The online, browser based interface keeps flagged messages in an easily accessible กquarantineก and allows the user to check for any false positives. White list functionality is provided to aide in the prevention of future false positives. A hosting company offering such an advanced service takes Spam and virus filtering for their customers seriously.

About The Author

Eric Lester has served for 4 years as the webmaster for ApolloHosting.com, http://www.apollohosting.com, and previously worked in the IT industry an additional 5 years, acquiring knowledge of hosting, design, and search engine optimization. Apollo Hosting provides website hosting, ecommerce hosting, vps hosting, and web design services to a wide range of customers.

This article was posted on August 16

by Eric Lester

Nextgen BitDefender Solution Adds Heuristic Spam

Nextgen BitDefender Solution Adds Heuristic Spam Filters

by: Roxana Danaila

BitDefender 8 SpamDeny, the new antispam solution for Windows workstations launched today is an advanced email protection tool which features updated, as well as new filters. ขI’m very satisfied with how this version has turned out, because internal testing has shown an improvement of over 35% in detection rates, over the previous version. I would like to take this opportunity to thank all our users and testers, as we wouldn’t have had such a great product without their hard work, input and feedback,ข said Nicolae Simon, BitDefender Product Marketing Manager.

The new rules of the updated Heuristic filter will dutifully trash some new scam spam types, such as Nigerian scams, online casino scams, lottery scams and fake university diploma offers, while further reducing the number of false positives.

Additionally, new filters have been included, such as the Image filter, which is able to recognize and tag images frequently used in spam email, the URL filter which blocks mail containing กsneakyก links, and the Charset filter, which will block any email written in กstrangeก characters.

Various bells and whistles, such as configuration wizards, better ways to manage your friends list, and a handy fix that will disable incoming mail notifications for spam email, so that Outlook will only signal when legitimate email arrives, complete the package.

The product is available for evaluation and purchasing as of today, and is priced at USD 19.95 for a single user license. BitDefender AntiSpam 7.2 users can, obviously, upgrade to BitDefender 8 SpamDeny free of charge.

About The Author

BitDefender™ provides security solutions to satisfy the protection requirements of todayกs computing environment, delivering effective threat management for over 38 million home and corporate users in more than 100 countries. BitDefender(tm) Antivirus is certified by all the major independent reviewers in the antivirus field ICSA Labs, CheckMark and Virus Bulletin.

Website: http://www.bitdefender.com

This article was posted on January 14

by Roxana Danaila

Are Your Emails Gettting Through?

Are Your Emails Gettting Through?

by: Charlie Cook

If sending email to a mailing list is part of your markëting or business building strategy, you want to make sure your messages are reaching your audience. Are your readers getting your emails? Due to the proliferation of sp@m, ISPs (your web access and email provider) have added content checkers to try to reduce the flow of unwanted mail. In principle this is a good idea. In practice, lots of sp@m is still getting through and many lëgitimatë emails, possibly including yours, are being blocked.
You may assume that since your content is lëgitimatë and because you only send messages to 0ptin readers, your prospects and clients are getting all of your emails. Bu presently sp@m filters are blocking emails that include words such as:

hëre
n0w
lëgal
questi0ns
lëgitimate
ordër
bu1k
chëck
mi11ions
st0ck
f0rm
markëting s0lution
rëmove
opp0rtunity
cl1ck hëre

Even to send this article to the people whoกve requested it, I’ve had to write carefully and creatively to disguise these words to get them through the sp@m filters.
In an effort to eliminate junk mail, most email providers have sp@m filters in place, and, smart as these are, they don’t know you are not a sp@mmer if it includes what it considers to be suspicious words. Some filters automatically dump suspicious email into a bu1k mail folder, others block them entirely.
Chances are you don’t consider yourself a sp@mmer. You don’t blindly send mi11ions of emails a day to people who don’t want your information. You do respond to client inquiries and send information to people who request it. Even if you only send email to people who have double optëd in for your ezine, you can have your newsletter or message labeled as sp@m if you’re not careful. 1030% or more of your emails may be getting filtered by overzealous ISPs.
Usually, use of just one กoffensiveก word won’t get your email blocked, but repeat it or use it in conjunction with another กoffensiveก word and your recipients won’t receive your articles or notes.
I hate junk email as much as the next person. But if the recipient has requested your ezine or an email response, they should get it. In there effort to eliminate sp@m, the ISPs are beginning to infringe on lëgitimatë communication and commerce.
What You Can Do
Review available lists of words that ISPs consider กsuspicioiusก. I’ve posted one list of some innocuous and common words that if used together or frequently can get your email blocked at: http://www.marketingforsuccess.com/wordstoavoid.html
Use one of the online content checkers to see which remaining words could create problems with sp@m filters such as:
Ezine Chëck http://ezinecheck.com/check.html
Lyris http://www.lyris.com/contentchecker/
Where possible, replace กoffensiveก words. If you need to use a word like กmarkëtingก because it’s the best word for the job, you can get creative with, as I did above, but some sp@m filters block emails containing odd characters.
No strategy for getting by the ISP filters is foolproof, but every effort you make to eliminate potentially problematic words will increase the likelihood that the people who want to read your emails will get them. Take a minute to check your email before you send it and you’ll increase the number of people who read it and respond to it.
2004 © In Mind Communications, LLC. All rights reserved.

About The Author

The author, Charlie Cook, helps service professionals and small business owners attract more clients and be more successful. Sign up for the Frëe Marketing Plan eBook, ก7 Steps to get more clients and grow your businessก at http://www.marketingforsuccess.com

[email protected]

This article was posted on July 13, 2004

by Charlie Cook

Not Knowing This Explosive Email Secret Can Kill Y

Not Knowing This Explosive Email Secret Can Kill Your Mailing List

by: Reginald Turner

How are you doing today? I hope you are having a very prosperous day. If you own an ezine or mailing list, you know how important it is to get subscribers, especially if you want to tell them about your products and services, or you just want to inform people in your mailing list about information that could help them. It could be very frustrating if it seems like your mailing list is not responding, right?

Let me ask you a couple of questions? Did you ever send out offers about your products and services to your mailing lists and it seems like your mailing list is not responding to it? Did you ever think that people in your mailing list wasn’t responding to your promotions because of your advertising? Last, but not least, did you ever try to rework your advertisements and the results you got from your mailing lists remained the same? Well I have some good news for you.

The products and services you offer, or the way you advertise could possibly have nothing to do with why it seems like people in your mailing is not responding. The reason it may not seem like people in your mailing list are not responding to your promotions could be because of the following reasons: spam filters, your blacklisted without even knowing it, or you are not whitelisted.

Most of the people who are either in your mailing lists or subscribe to your mailing lists will probably belong to the following major ISP’s: AOL, NetZero, Yahoo!, AT&T, Msn/Hotmail, Mail.com, Earthlink, and Comcast. The majority of these major ISP’s use high tech spam filters, blacklists and look to see if you are whitelisted.

If you are into email marketing, but not familiar with terms such as spam filters, blacklists, and whitelisting, you might be asking yourself, what do they mean? how could they hurt me? For your benefit, I will be more than happy to explain to you what they mean and how they could hurt you, especially if you own a mailing list.

First, lets talk about what blacklisting is. Blacklisting is a list of spammer IP addresses that the majority of the major ISP’s keep. If your email server IP address is found on their blacklists, it could stop your message from being delivered and that could hurt you if you’re trying to promote your products and services to your mailing lists. The scary thing about blacklists is you don’t even know if you are on them or not.

Second, lets talk about spam filters. Spam filters is a piece of software that the majority of the major ISP’s use to determine which emails are spam and aren’t spam. Lets say you deliver an email to somebody in your mailing lists that has an email account with Netzero. Before that email even gets to their inbox, it has to pass through their spam filter. If there are any words in your email that causes their spam filter to trigger, it will consider your email as spam and block your email from getting through. The scary thing about this is there is know way of telling what words will trigger the spam filters when you are writing your emails. This another factor that could hurt you, especially if you own a mailing list.

Last, but not least, lets talk about whitelisting. Whitelisting is a process of telling people in your mailing list to add your ezine to their “safe” or permitted email senders lists. Although this process could be very beneficial, it could also be very extensive and time consuming, especially if you have a huge mailing list. However, not being whitelisted could also hurt if you own a mailing list.

If you are like me and have a mailing list, you don’t want to have to deal with spam filters, blacklists, or the process of getting whitelisted. For more information on how to beat these mailing list killers so that your message gets to your mailing lists, please visit: http://www.adalyzer.com/727/email_info. Thank you very much for taking the time to read this article and I hope it benefits you to the fullest.

About The Author

My name is Reginald Turner and my purpose in writing articles is not only to help internet marketers become more efficient and smarter when it comes to marketing over the internet, but to teach them how to quickly find out what type of marketing works and how to make more money over the internet in less time while exploding traffic to their website. If you would like more info about website marketing techniques, you could just visit: http://www.adalyzer.com/727/traffic_attack.

[email protected]

This article was posted on April 13

by Reginald Turner

Which Spam Filter Is Best For You?

Which Spam Filter Is Best For You?

by: Niall Roche

With the number of spam filtering solutions increasing each week itกs getting tougher for consumers to make informed choices in their purchases.

There are 3 basic types of spam filters:

Integrated

Standalone

Online

Weกll look at each type of spam filter and at the end you should be able to decide what spam filter is right for you.

Integrated spam filters

This type of spam filtering software is the most common. Once installed it sits กon topก of your existing email software and installs a new set of buttons into your email software. In future when you collect email youกll see options for marking email as Spam, marking the email as Not Spam, Bounce the email back to sender, etc. The description and position of these buttons varies from one product to the next but their purpose remains the same.

Most integrated spam filters automatically place suspected junk email into a separate folder on your PC for you to review or delete later on.

The newer integrated spam filters are also กintelligentก. They can basically learn the difference between what is spam and what is not and delete the junk email you don’t want.

The most popular integrated spam filters are:

iHate Spam

Spambully

Spam Inspector

Integrated spam filters are most popular amongst people who want a one click solution to collecting their personal email and filtering junk email at the same time.

Advantages:

One click solution.

Disadvantages:

Software specific. Some work with Outlook and Outlook Express only.

Standalone spam filters

These are less common than their integarted counterparts but that doesn’t make them any less useful. A standalone spam filter is basically a separate piece of software installed on your PC that you use to check your email for spam.

Standalone filters have the big advantage of being able to preview your email on the mail server before itกs downloaded to your PC. This one single feature has the huge benefit of allowing you to just download the email that you want as opposed to downloading all of your email, including the spam, and then sorting through it.

Using a standalone spam filter is a little more work simply because itกs a separate piece of software that you have to run before you open up your email software. Most standalone filters do allow you to configure them so that your standard email application is opened once youกve chosen what spam to filter. This suits some people and not others.

The most popular standalone spam filter is:

Mailwasher Pro

Advantages:

Doesn’t rely on specific email applications to work properly.

Disadvantages:

Two step process. Load standalone filter and then your email application.

Online spam filters

There are really two types of online spam filters. One is for business use and one is for home use. A typical example of a business type product is iHate Spam server edition where the software deletes junk email directly from the mail server before the end user even sees it. Large companies employ this type of technology.

Home users will be using Spam Arrest or similar. Spam Arrest offers an inventive solution to spam whereby any email sent to the users account has a challenge request sent back to it which the sender must authenticate. The automatic junk email software used by spammers can’t currently deal with this type of response. Any failure to authenticate the challenge email results in the junk email being left to die in cyberspace. A user is authenticated with Spam Arrest only once for security just to make sure the software doesn’t become a nuisance.

The most popular online spam filter is:

Spam Arrest

Advantages:

Users are guaranteed to only receive the email that they want or requested.

Disadvantages:

Any techncial problems with the Spam Arrest server and you have no defense against spam.

Now youกve seen what spam filtering options are available to you just ask yourself which one suits you most. If you’re still not sure drop by http://www.spamsite.com and check out our product reviews we have something for everyone.

About The Author

Niall Roche is the content author and owner of http://www.spamsite.com

[email protected]

This article was posted on February 23, 2004

by Niall Roche

Is Your Ezine Being Zapped?

Is Your Ezine Being Zapped?

by: Michael Southon

About a year ago I wrote an article titled กWinning The War On Sp^mก. Unfortunately, the war on sp^m is not being won at all.

In fact, the problem is now so serious that sp^am is shaping up to be the greatest threat to online marketing.

The threat comes not from sp^mmers themselves, but from the filters that are being used to block them.

These filters are hitting hard at the very core of ecommerce Ezine Publishing.

Antisp^m filters operate at two levels: (i) clientside programs that reside on individual computers and (ii) serverside programs that ISPs are using to block incoming sp^m.

The problem is that the filters are now so sensitive they are blocking even the most innocent of Newsletters.

For example, if your Newsletter contains the words ‘removeก, กunsubscribeก or กclick hereก it will trigger antisp^m filters in many of the programs that are now being used by ISPs.

The result?

Your Ezine is zapped, deleted and a large percentage of your subscribers will think you have stopped publishing your Newsletter.

What can you do about it?

Here are some tips to avoid sp^m filters:

Post your Newsletter online and then email your subscribers to tell them that the latest issue is now available online.

In your Newsletter carefully avoid (both in the subject line and the body text) all words that are likely to trigger antisp^m filters. Use the free service listed at the end of this article it will flag any words in your Newsletter that trigger antisp^m filters.

Instead of saying ‘to unsubscribeก (which is a phrase commonly found in sp^m), say กIf you no longer wish to receive…ก or กIf you wish to leave this mailing list…ก or ‘to take yourself off this list…ก

If there are trigger words that you simply cannot avoid, you can disguise them using carets (^) or other symbols. The กFก word would become fr^e and the กUก word would become uns^bscribe.

Include the word กNewsletterก in the subject line of your email this will help the filters identify your email as nonsp^m.

Avoid whole words in upper case. In many Newsletters the headers are capitalized this will trigger the filters.

If your Newsletter contains ads, scrutinize them carefully ezine ads, by definition, contain words frequently used by sp^mmers.

Here is a fr^e service that will help you avoid sp^m filters. Before you mail out your Newsletter, just send a copy of it to the email address below with TEST in the subject line: mailto:[email protected]

Within a few seconds youกll receive a report that analyses your Newsletter and gives you a score (0 to 5=no problems 1216=over the limit for most ISPs).

If you write articles, itกs worth submitting them to this test as well, together with your Resource Box (I just sent this article to Sp^mCheck and got a score of 4.6).

Sp^mCheck is operated by Sp^mAssassin, a filter that is widely used by ISPs so this is a good test of whether your Newsletter will get through to your subscribers.

© 2002 by Michael Southon

About The Author

Michael Southon has been writing for the Internet for over 3 years. He has shown hundreds of webmasters how to use this simple technique to get massive free publicity and dramatically increase traffic and sales. To find out more, please visit: http://www.ezinewriter.com

This Article may be freely published in its entirety exactly as it appears above. No alterations or changes to the Article are allowed, without the express permission of the Author. The Resource Box must remain with the Article just as it appears.

This article was posted on December 10, 2002

by Michael Southon

Keeping Your Email Box Organized & Ready for Busin

Keeping Your Email Box Organized & Ready for Business

by: Stone Evans

Learning how to get the most organization from your email software is very likely the most important skill you can learn to conduct a successful online home business.

There are three primary email applications on the market that folks use to manage their incoming POP3 email. Those three are:

Microsoft Outlook Express

Eudora

Netscape Mail

In order to get the most from your email software, there are three key processes that you should learn. These processes are concerned with data organization, saving time and email database management.

SET UP EMAIL FOLDERS

Organization is key to any emails that you intend to save. Having 2000 emails in one folder is a surefire road map to confusion and lost communications and information.

You are the best judge as to how to organize your email into topics that provide an easy method of retrieval of the information when you need it most.

Fortunately, the primary email browsers make it easy to organize your information. By allowing you to create folders within your email software, you can file specific emails into folders dedicated to the topic of the email.

To create new folders:

OUTLOOK EXPRESS: 1. Click File Folders New alternatively, you may also right click an item in the Folder window and then type in the name of your new folder. Highlight the folder above where you would like your new folder to be placed, and then click OK.

EUDORA: Click on Mailbox New or right click on Eudora in the folders window and then click on New. When the window opens, type in the name of your new mailbox and click OK. If you want to create a folder to place other mailboxes into, click the checkbox before clicking OK.

NETSCAPE MAIL: Click File New Folder. Then from the dropdown menu, select the folder that you wish to be the folder directly above your new folder, before clicking OK.

FILTERING EMAIL

Email filters are a tool to help you save time and frustration. Have you ever lost an incoming email under the deluge of email coming into your mailbox? With filters, you can direct the important email or notsoimportant email into certain preordained folders.

As part of the war against spam, most ISPกs use filters every day in an attempt to keep the spam out of your mailbox. In fact, I even use filters to sort my incoming mail trying to catch the spam my ISP missed.

Don’t be fooled into thinking you need some special software to filter your email. Setting up filters is actually quite easy.

To setup email filters:

OUTLOOK EXPRESS: Click on Tools Message Rules Mail. A wizard will open to help you create your new email filters. Just follow the instructions provided to direct mail based on certain criteria into certain folders.

EUDORA: Click on Special Make Filter. Once again, just follow the instructions in the Filter wizard.

NETSCAPE MAIL: Click on Edit Mail Filters. Then click on New in the wizard. Once you click on New, you will be taken to a new wizard window. Follow the instructions here, then when you are done, click on OK. This will return you to the first window where you will set up the sorting order of your filter.

CLEANING AND COMPRESSING YOUR MAIL DATA

This is an important part of your email management. When you no longer need an email, it should be deleted. When you first delete an email, your software will send the email to the Trash Bin. Your email is not actually deleted until you first empty your trash bin.

Emptying your trash bin compresses the mailboxes from where the email was originally filed. This is absolutely paramount to the protection of your email data. If you go too long between compressing your email data, then your email data could become corrupted and you might need assistance in recovering your email data.

Even after you have emptied your trash, Compressing Folders is a recommended step to prevent other data corruption. Once you understand that an email does not actually move from one folder to another until the folder is compressed, then you can better appreciate this advice.

As an example, when Email A comes into your main Inbox, the data connected to Email A appears in two files. One file contains the header and body of the email. The other file contains only the email header information.

When an email is moved from one folder to the other, only the header information is actually moved. The body information will not be deleted from the original folder until which time the original folder is compressed.

This explains the purpose of emptying the trash AND compressing folders. If the email was simply moved from the Inbox to another folder, then emptying the trash is not enough. The original placement of the email is not actually removed from the file that contains the body information until the message has been designated for compression.

To empty your trash:

OUTLOOK EXPRESS: Click on Edit Empty กDeleted Itemsก Folder. To compress the remaining folders, click on Tools Options, then click on the tab for Maintenance. Then click the button that says, กClean Up Nowก. Once the compression is completed, click OK.

EUDORA: Click Special Empty Trash. To compress the remaining items that need compression, then click on Special Compact Mailboxes.

NETSCAPE MAIL: Click on File Empty Trash Folder. To compress the mailboxes, click on File Compress Folders.

I cannot stress enough how important it is that you utilize the tools for emptying the trash and compressing the mail folders. Protecting your email data on a regular basis is good practice for avoiding disaster in your mailbox.

So many of us rely upon our email software to keep our online business running smoothly. Once you master the tools provided in your email software, your online business will run smoothly also.

Let me share one important lesson I lกve learned about computer software. One should never be afraid to try new things. Learning how to get the most out of your software relies upon your willingness to dig in and learn how to use it.

Once you learn how to use your software to its full potential, then your life will be greatly simplified and your effectiveness will be dramatically improved.

About The Author

Stone Evans owns the Home Business Resource Directory where you can find everything youกll ever need to start, run and grow a home based business at: http://www.HomeBusiness.com

[email protected]

This article was posted on March 19, 2004

by Stone Evans

Spam Filters Explained

Spam Filters Explained

by: Alan Hearnshaw

What do they do? How do they work? Which one is right for me?

Spam is a very real problem that many people have to deal with on a daily basis. For those that have decided to do something about it and start to investigate the options available in spam filtering, this article provides a brief introduction to your options and the types of spam filters available.

Despite the bewildering array of spam filters available today, all claiming to the best one ขof its kindข there are really just five filtering methodologies in general use today and all products rely on one, or a combination of these:

ContentBased Filters

ขIn the beginning, there were contentbased filters.ข

These filters scan the contents of the and look for telltale signs that the message is spam. In the early days of spamming it was quite simple to look out for ขKill Wordsข such as

ขLose Weightข and mark a message as spam if it was found.

Very soon though, spammers got wise to this and started resorting to all kinds of tricks to get their message past the filters. The days of ขobfuscationข had begun.

We started getting messages containing the phrase ขL0se Welghtข (Notice the zero for ขoข and ขlข for ขiข) and even more bizarre – and sometimes quite ingenious – variations.

This rendered basic contentbased filters somewhat ineffective, although there are one or two on the market now that are clever enough to ขsee throughข theses attempts and still provide good results.

Bayesian Based Filters

ขThe Reverend Bayes comes to the rescueข

Born in London 1702, the son of a minister, Thomas Bayes developed a formula which allowed him to determine the probability of an event occurring based on the probabilities of two or more independent evidentiary events.

Bayesian filters ขlearnข from studying known good and bad messages. Each message is split into single ขword bytesข, or tokens and these tokens are placed into a database along with how often they are found in each kind of message.

When a new message arrives to be tested by the filter, the new message is also split into tokens and each token is looked up in the database. Extrapolating results from the database and applying a form of the good reverend’s formula, know as the a ขNaive Bayesianข formula, the message is given a ขspamicityข rating and can be dealt with accordingly.

Bayesian filters typically are capable of achieving very good accuracy rates (>97% is not uncommon), and require very little ongoing maintenance.

Whitelist/Blacklist Filters

ขWho goes there, friend or foe?ข

This very basic form of filtering is seldom used on its own nowadays, but can be useful as part of a larger filtering strategy.

A ขwhitelistข is nothing more than a list of email addresses from which you wish to accept communications. A whitelist filter would only accept messages from these people and all others would be rejected

A ขblacklistข, conversely, is a list of email addresses and sometimes IP Addresses (computer identification addresses) from which communications will not be accepted.

While this may seem like a good idea from the outset, a whitelist methodology is too restrictive for most people and, as virtually all spam emails carry a forged ขfromข address, there is little point in collecting this address to ban it in future as it is very unlikely to be the same next time.

There are bodies on the internet that maintain a list of known ขbadข sources of email. Many filters today have the ability to query these servers to see if the message they are looking at comes from a source identified by this Internetbased blacklist, or RBL. While being quite effective, they do tend to suffer from ขfalse positivesข where good messages are incorrectly identified as spam. This happens often with newsletters.

Challenge/Response Filters

ขOpen sesame!ข

Challenge/Response filters are characterised by their ability to automatically send a response to a previously unknown sender asking them to take some further action before their message will be delivered. This is often referred to as a ‘turing Testก named after a test devised by British mathematician Alan Turing to determine if machines could ขthinkข.

Recent years have seen the appearance of some internet services which automatically perform this Challenge/Response function for the user and require the sender of an email to visit their web site to facilitate the receipt of their message.

Critics of this system claim it to be too drastic a measure and that it sends a message that กmy time is more important than yoursก to the people trying to communicate with you.

For some low traffic email users though, this system alone may be a perfectly acceptable method of completely eliminating spam from their inbox one step above the กWhitelistก system outlined above.

Community Filters

ขA united frontข

These types of filters work on the principal of กcommunal knowledgeก of spam. When a user receives a spam message, they simply mark it as such in their filter. This information is sent to a central server where a ขfingerprintข of the message is stored.

After enough people have ขvotedข this message to be spam, then it is stopped from reaching all the other people in the community.

This type of filtering can prove to be quite effective, although it stands to reason that it can never be 100% effective as a few people have to receive the spam for it to be ขflaggedข in the first place. Just like its similar cousin the Internet black list (RBL), this system also can suffer from ขfalse positivesข, or messages incorrectly identified as spam.

Hopefully you are now armed with a little more information to be able to make an informed decision on the best spam filter for you.

For further information, consider reading the reviews and articles found at http://www.whichspamfilter.com

About The Author

Alan Hearnshaw is the owner of http://www.whichspamfilter.com, a web site which conducts weekly indepth reviews of current spam filters, provides help and guidance in the fight against spam and provides a useful community forum.

[email protected]

This article was posted on November 08, 2004

by Alan Hearnshaw

So What Makes a Good Spam Filter Anyway?

So What Makes a Good Spam Filter Anyway?

by: Alan Hearnshaw

Spam Filters. Most of us know we need one. Some of us know we need a better one, but how many stop to think what actually makes a good spam filter in the first place?

This is not just a rhetorical question. It is a question that many users – and many developers do not ask, and consequently, it largely remains unanswered.

Maybe this could be better answered by defining here the qualities of the perfect spam filter. We’ll call our perfect spam filter the ขSpamSplatter 3000ข. Here are some of the defining qualities of ขSpamSplatter 3000ข

It requires zero interaction from the user.

It produces zero false positives (good messages identified as bad) and zero false negatives (bad messages identified as good).

It is transparent – that is, you only ever see good messages and never need even be aware that spam exists.

That’s it. Not much of a shopping list is it?

Of course, ขSpamSplatter 3000ข hasn’t been invented yet (and if it does, I want a piece of the action), but it does give us a frame of reference when looking for the best filter we can find.

Let’s take each point in turn:

It requires zero interaction from the user

There are two kinds of filters that come near to this ideal currently: Bayesian Filters and Community Filters.

Bayesian filters strip messages down to small ขword bitesข, or tokens and maintain a database containing lists of good and bad tokens. When a new message is encountered, the filter strips this message down to tokens, compares it to the database, and applies a formula based on the British scientist Alan Bayes’ formula for probability calculation.

Over time, the Bayesian filter ขlearnsข the characteristics of spam messages.

Community Filters simply work on a voting system whereby every user that receives a spam message ขvotesข it as spam. This information is stored on a central server and when enough votes are received the message is banned from all users in the community.

As can be seen, the user interaction from these types of filters is mainly limited to two button operation – correcting wrongly identified messages – and the more accurate the filter, the less those buttons are used.

OK, so that’s pretty good. Not exactly zero interaction, but if the filter is accurate enough, then it should be pretty near. That brings us to point two:

It produces zero false positives or negatives

This is the area in which most spam filter development is concentrating and things are getting pretty good nowadays. It is not at all unusual to see an efficient modern filter achieve accuracy of 96% or better. It is, of course, far better to have a false negative than a false positive if you are ever going to tear yourself away from the killed mail folder!

Of course, by definition, community filters cannot reach 100% accuracy as someone has to be getting the spam to be voting it as such!

Theoretically, a Bayesian filter may be able to eventually get quite close to 100% accuracy, so at least there is hope there.

Content based filters (those that look for certain words, phrases or other indicators in a message to identify it as spam), will almost certainly not get much higher accuracy figures than the best of them can achieve today. Adapting to changing spam requires new filters to be created on an ongoing basis.

And finally, we come to the holy grail of spam filtering:

It is transparent

Strangely enough, not enough work seems to be done in trying to achieve this goal. Some of the best filters on the market today identify spam with impressive accuracy and then simply place them in a ขkilled mailข folder for your later perusal.

Now, forgive me if I’m missing something here, but isn’t the point to save you having to wade through the junk mail? Isn’t that what you bought the filter for? With the ขSpamSplatter 3000ข, you don’t need to do that.

As we haven’t achieved 100% accuracy yet (and probably never will), the only way to free us from checking the killed mail folder is a challenge/response system. This is where a message is automatically sent back to the sender requiring them to take some action for their message to actually be delivered.

Some systems tend to go overboard with the challenge/response system. These systems often called ขWhitelistข systems block messages from anyone that isn’t in the user’s friends list. Guaranteed 100% effective, but too drastic a measure for most users.

Now, it seems that the most intelligent use of this system would be to send challenges only to messages that were flagged as ขquestionableข. Good message can be delivered, definite spam can be deleted and questionable ones would earn themselves a challenge message.

So, to sum up, let’s rewrite the qualities of our perfect filter and get a shopping list of what to look for while we wait for the ขSpamSplatter 3000ข to arrive:

Simple, minimal setup and maintenance.

Extremely low rate of false positives and as few false negatives as possible.

A transparent ขfailsafeข mechanism whereby the victims of those false positives can force the message through to you.

It’s simple really. Now, who’s going to build me this ขSpamSplatter 3000ข…?

About The Author

Alan Hearnshaw is the owner of http://www.WhichSpamFilter.com, a site which provides weekly indepth spam filter reviews, antispam help and guidance, user ratings and a community forum.

[email protected]

This article was posted on November 08, 2004

by Alan Hearnshaw

Spam Filters & Blockers: The Solution To Unsolicit

Spam Filters & Blockers: The Solution To Unsolicited Email?

by: Anti Spam League

Spam is a term that, for the vast majority of us, has a very negative connotation. How could it be different when the average Internet user gets over 50 pieces of junk email per day? Moreover, research indicates that by 2006 the average internet user can expect to receive at least 5,000 pieces of spam per year. That sure is a giant amount of spam!

None of us want to waste our time, our bandwidth and our hard drive space to receive spam and then delete it. Because our daytoday lives are delayed, interrupted and significantly disturbed by this large number of unsolicited email, software manufacturers have developed a number of products especially designed to fight spam. These products might be called ‘spam filters’, ‘anti spam filters’, ‘email filters’, ‘spam blockers’ and even ‘spam killers’. Regardless of the name, their purpose remains the same: to eliminate the spam in your inbox. We have not yet seen a spam filter that achieves 100% accuracy but the latest products launched into the market as well as the newest versions of existing products are proving to be more and more successful.

In case you are wondering specifically what these products do and how they do it, you should know that a spam filter is a program that web based, server based or installed locally prevents spam email from being downloaded to your PC. The spam filter itself operates on the basis of rules, which means that the spam fighting program will examine your incoming email and match it against a set of predefined criteria. If the email does not match those rules, then it is either: a) Deleted, or b) Quarantined for Review. There are three basic types of traditional antispam software:

1) Plugins for email clients These anti spam programs run on your computer and require all received spam to be downloaded before they start sorting it.

2) Standalone applications These spam blockers also run on your computer and communicate with the email server multiple times each hour to check your POP mailbox for spam and delete it. They constantly run in the background, using up resources and bandwidth.

3) Serverbased email spam filters Many ISPs and email servers (Hotmail, Yahoo! Mail, etc.) have installed email filtering software that scans email for certain triggers, which may include certain phrases, formatting, and aggressive writing styles. A server based spam filter is designed to examine the email being sent to your Internet Service Provider or local area network and to filter it effectively before the email is delivered to you. When this works, it works effectively, however it is not perfect. Unfortunately, many unfair marketers have learned to fool spam filters while a lot of legitimate emails are blocked by them.

Of the spam filters that are on the market at the moment the vast majority of them need to be installed to your hard drive. However, there is a move towards spam filters that do all the work online before the email ever gets to your computer. Most of today’s email applications come with a basic spam filter that allows you to block email from specific addresses. Some more complicated email software allows you to configure rules just for that software itself. This can get pretty complex pretty quickly! But people who suffer from tons of spam want a simple solution to their problem not more headaches. On this note, one of the most popular PCbased spam solutions on the market today is Mailwasher Pro, which has a really nice feature that allows you to send a reply to the spammer that mimics a ‘mailer daemon’ error message; that is, when the spammer gets this message from you he assumes your email address no longer exists. Simple but effective.

Other wellknown spam blockers are iHate Spam, McAfee Spamkiller, Spam Inspector 4.0, Spam Arrest, Qurb, Choicemail One, among others. iHate Spam is an easytouse and effective spam filtering solution for Outlook, Outlook Express, Eudora, Incredimail and Hotmail users. McAfee Spamkiller has several innovative features such as protection against Dictionary Attacks (made up names) and foreign language spam, and also supports Outlook, Outlook Express and other popular email programs. Based on personal and global learning networks, Spam Inspector 4.0 adapts itself to your email automatically, filtering out the junk mail with an impressive accuracy.

A particular type of spam filters are the socalled ‘Bayesian spam filters’. These are considered intelligent filters, because they are capable of comparing two sets of information and acting on the result. This differentiating feature of Bayesian filters is in direct opposition with the vast majority of other spam filters which use prebuilt rules to decide which email is spam and which is not. Bayesian spam filters take one group of legitimate email and another group of spam, and then compare the values and data of each. They look for obvious repeating patterns to form an ‘opinion’ on something. In spam filter terms that ‘opinion’ becomes a rule. The definition of legitimate email that they create at the end of this comparison session is what they use going forward to scan your inbox for spam. Yes indeed, these spam filters are capable of learning. For example, if the filter blocked an email because it was perceived as junk, then the user marked it as a valid email, then t!

hey will not to block that type of email in the future. Thus, in time, this type of spam filter learns enough to block spam far more effectively. Some of the Bayesian spam filtering options available today include AOL´s Communicator product, Eudora, and Spambully. Although the latter is a fairly new product, it is already making waves in the sea of spam fighting software. Spambully provides an integrated solution for users of both Outlook and Outlook Express, and its creators claim it can remove 99% of spam.

Finally we would like to point out two free spam blockers that provide good filtering capabilities. They are prime examples of Bayesian spam filters and perfect examples that freeware does not necessarily mean low quality. The first of these products is called Popfile and the second one is called Spampal. Both are Windows only software products.

In conclusion, although no spam blocking product can currently guarantee 100% accuracy, one thing is for sure: the more positive, affirmative action you take against spammers by deleting, blocking and/or blacklisting them, the less spam you will receive. Spam filters and spam blockers can really help us along the fierce battle against spam. The more action we take, the smaller the spam problem will become. Wouldn’t that make us happier?

If you want to learn more about this and other related topics, check out www.AntiSpamLeague.org. This organization offers free membership and the chance to access a wide amount of relevant information on privacy, spam, email abuse, Internet fraud, responsible marketing and several other topics.

About The Author

The purpose of the Anti SPAM League is to help consumers and business owners reduce the amount of SPAM they receive. In addition, our Anti SPAM organization believes that educating site owners in the area of SPAM prevention and ways to successfully and responsibly market their sites, is key in making a difference.

[email protected]

This article was posted on December 22, 2004

by Anti Spam League

What Your Car Audio Can Teach You About Marketing

What Your Car Audio Can Teach You About Marketing

by: Mike Street

Stand next to any road, and every so often a young person will go by with the latest rap CD blaring. If it happens to be a cold day, he (it is always a he) may have the windows up. Then, all you will hear is the thud of the overworked bass speaker in the back. After he turns 30, the young driver probably won’t even be able to hear that, if he continues this unwarranted assault on his ears.

That big bass bin can’t handle the vocal sounds, and the front speakers would melt if they had all those thumps going through them. So the car audio separates out the various frequencies using filters, sending only the bass to the big bins in the back, and only the higher and more delicate sounds to the little speakers at the front. Both, and especially the bass, are then amplified so they are audible in the next county.

Marketers have borrowed the same terminology as a way of looking at how their business treats its customers. Marketing graduates will often talk about ‘Filters’ and ‘Amplifiers’ almost as if they actually understood them.

Filters

A filter in marketing speak is anything which prevents your customer from doing business with you. Some filters are ‘natural’ – if you provide personal training services for example all of your clients will need to be within easy reach. This natural, geographic filter means that you are unlikely to sell to someone in another country.

Others are contructed. Mercedes Benz dealers the world over have large, bright, glassy establishments. They tend to intimidate anyone who can’t afford the prices, acting as a natural, probably intentional, filter.

Filters can also be fairly subtle. If you send out a mailing by post, research shows many are discarded without even being opened. If you have a leaflet delivered, the ‘open an envelope’ filter is removed, so people can’t help reading it, even while they are trying to throw it away.

Amplifiers

An amplifier is anything which increases the ease of doing business with you. Any business which decides to take payment via credit cards, for example, will find the number of people who can do business with them is amplified compared to when only cash was acceptable.

Marketers probably won’t admit it, but filters and amplifiers are opposites of the same thing. Removing a filter has an amplifying effect, and vice versa.

I insured my car the other day over the Internet. The first few sites I tried only supported Internet Explorer. That, at least to me, is a filter and I went somewhere a little more Firefox friendly!

Many corporate web sites insist you provide a lot of information before they will send you that ‘free’ White Paper you are interested in. No doubt that information is required by someone in the business, but it filters out a lot of otherwise interested people who simply won’t take the time to fill out the form and inevitably receive all the sales calls afterwards. After all, they can’t be sure they’re even a prospect before they read the White Paper!

Apple has potentially filtered out a large portion of their target market for iTunes by only accepting credit cards. Most under 18s won’t have a credit card, and they are the major buyers of chart music. The ‘Music Store Card’ is an attempt to turn this filter into an amplifier.

What Filters and Amplifiers Mean to Your Marketing

Importantly, this way of thinking allows you to look at all of your marketing, online and offline, in a critical way to improve your response rates and your sales. Every time you look at any aspect of your business, ask yourself if this filters out customers you want to serve, or if you can amplify the target market by improving the process.

Perhaps you could send postcards or use leaflets instead of putting brochures in envelopes. Don’t insist on a customer’s life history before you will allow them to buy from you. Make your web site informative and easy to use, rather than slick, ‘cutting edge’ and hard to understand. If you are providing services, make it clear on your site where you are and the distance you will travel. Use local town and county names as keywords to filter out people who will never be able to buy from you, but to amplify the chance of attracting locals.

If you do this consistently, over time you will get your filters and amplifiers to attract profitable customers to you, not send them away to your competitors, never to return.

About The Author

After more than 30 years in the IT industry, Mike Street is a director of FastComm (http://www.fastcomm.net) which specialises in information and tools to help increase sales, including Airlook Mobile Email software, the Eye Catcher Video Phone and the online Contact Management system, FastCRM. He is also webmaster of his wifeกs Health and Beauty site Zenergie (http://www.zenergie.co.uk).

[email protected]

This article was posted on April 23

by Mike Street