Sequence of Plugin Filters

[Zaxon]

<< Updated 21.12.08! >> (~Chactory)

There have been many questions about what order you should place the plugins in with Spami. Bob, and many others, have given out some very good advise throughout the forum. I want to distill the overall advise into one thread, and in particular, add my own additional rules onto what has already been discussed so far.

The general rule of thumb is:

1) Filters which detect non-spam only go highest/first - everyone agrees on that

2) Second, you'll find two schools of thought come out of the forum:
School 1:
* Filters which detect spam only go second, followed by filters which classify emails into spam and non-spam going last
School 2:
* Filters which classify emails into spam and non-spam go second, followed by filters which detect spam only going last

You'll find threads espousing both schools of belief in different threads already, so I won't go over that.

Filter Sequence - according to Zaxon

I'll add some extra things to consider when prioritizing your filters. This doesn't so much contradict what I've stated above, rather than to put a slightly different spin on it.

1. Filters which classify non-spam and which you believe are 100% safe go at the top

This is almost the same as the general rule above, except that I have added the "and which you believe are 100% safe" portion. Not all non-spam only filters are necessarily right for this category. See the safety concern below.

2. Safety Filters go first

There are certain filters that actually protect your safety. These are filters which look for harmful scripts, web bugs which actually report back that you've opened an email and therefore confirm your email address etc. These are real safety concerns, whereas emails which contain the phrase "buy * POLICY VIOLATION ! *" might be annoying, but they don't actually pose a threat.

Filters that contain safety checks are: Scripts Filter, Attachment Filter, Hercule Filter, Image Filter (web bugs), and there may be some others.

For instance, you probably don't want a message about your favourite pop star or TV program if, unbenowns to you, it also will run an Active-X script, and report back to base that you've opened it. It fact, I found out that several of the newsletters I was reading reported back to base. Thank you Hercule Filter for alerting me to this. So, I've chosen to block all mail that does this.

3. The filter type which is most accurate goes next

Assuming you have the Filter Statistics Plugin installed, you'll find from the [General Statistics] the figures for False-Positives and False-Negatives.

It makes a world of sense to put the filter type that has been the most accurate for you, first.

So if your False-Positives % is very low and your False-Negatives % is far higher, then put the filters which classify spam first (blacklist, DCC etc), and your less accurate filters which classify non-spam (learning, URL) last.

If your False-Negatives % is lower than your False-Positives %, then the filters doing the most accurate job are your Non-Spam detecting filters or Spam/Non-Spam filters (Learning, URL). Put your less accurate filters that classify spam-only (blacklist, DCC etc) last.

You do this because, like me, you insist on spami giving you the most accurate results possible. So honour the most accurate filter type by promoting filters of that type up the list, and the less accurate filter type by placing them down the list.

4. Filters which give false-positives get an instant demotion

You know if you were to have a noisy, personal phone conversation at work, people may raise their eyebrows, but if you started stealing money from the company you'd get fired?

False Negatives, spam which isn't recognized but is let through, is something to raise your eyebrows about.

False Positives, good emails that get stopped and labelled as spam, means that somebody deserves to be summoned to the principal's office to get the strap. Somebody has to * POLICY VIOLATION ! *!

If a filter incorrectly marks good emails as spam, instantly demote it down your priority list. You want to stamp out this behaviour as quickly as possible. Note, some filters are configurable. So, delete that blacklist that triggered a false-positive. If 10 <html> tags caused a spam rating in Hercule on a good email, up it to 20, etc. When a filter reaches the bottom of the priority list and stays there for a while, meaning it's consistently inaccurate, it's probably time to permanently delete it.

5. Highly effective filters get promoted up the list

If one of your filters catches 60% of spam, while another only catches 5%, promote the 60% filter up the list. Why? Because you should have the filters which catch the most spam first in line, so that spam is caught as soon as possible. If an email does reach the lazy filter that catches only 5%, then its a rarer occurance. You want spami to run as quickly as possible, so catch the spam emails as high up in the filter chain as possible. Remember, spam which is caught stops immediately, and doesn't have to pass through any of the subseqent filters.

And if you're feeling sorry for those filters that only catch 1 or 2 percent, and they tell you, with there big puppy dog eyes, that "if only they were given the chance, they could be a star performer too..." This might well be true, but it's better to have one filter catching 90% of spam than 9 filters catching 10% of the spam each.

6. Lazy filters get sacked

If you find that there are filters that never seem to catch anything over a period of time (months), give them the sack - delete them. Why? Because we all want spami to process mail as quickly as possible, if a filter is failing to perform on a consistent basis, you should delete it. Think of all the collective minutes you have waited for emails to be processed by that filter, that doesn't seem to be catching any spam anyway. Perhaps you have redudant filters: a filter higher up in the list that's already doing the job. Or perhaps you have a poorly coded filter that doesn't seem to match the author's expectations when placed in the real word. Say to your lazy filters, "You're fired!"


That's about it. Every experienced user of Spami will have their own believes on how filters should be ordered. These are mine. I've explained them particularly, because they add a bit more insight beyond the common whitelist > grey lists > black lists way of looking at things.

Anyway, something to think about.

[Gast]

Zaxon

Thanks for that.

Could I ask you to list your filters in order.

I'm willing to bet there are a lot of people like me that want a standard product to highlight spam. I'm not too worried about getting that extra 1% accuracy at this moment in time - I'm happy to kill spam: full stop.

I need somebody to say I use these plug-ins in this order and it works for me. As time progresses and I get used to the programme I will make changes myself. For instance, how do I know which filters work the best??

Spami has a great reputation. However, as a new user to be offered a big choice of filters and not being sure what to do probably loses Spami some users. It's nice to have choice when you know what you are doing, otherwise, you go with a product that can't be customised and make do.

Thanks
Ian

[Zaxon]

Thanks for that.

Could I ask you to list your filters in order.

I'm willing to bet there are a lot of people like me that want a standard product to highlight spam. I'm not too worried about getting that extra 1% accuracy at this moment in time - I'm happy to kill spam: full stop.

I need somebody to say I use these plug-ins in this order and it works for me. As time progresses and I get used to the programme I will make changes myself. For instance, how do I know which filters work the best??

Spami has a great reputation. However, as a new user to be offered a big choice of filters and not being sure what to do probably loses Spami some users. It's nice to have choice when you know what you are doing, otherwise, you go with a product that can't be customised and make do.

The spirit of this thread was to give people a way of looking at their own spam history, and giving them guidelines on how to tailor their filter order to their changing circumstances.

However, I can also appreciate that there are lots of users of spami who are new, and so have no history to draw on. And I can appreciate that there are lots of users who don't understand, as yet, exactly what all the filters do. And so, a recommended list of filter priorities could be useful.

So, with that in mind, I'll provide you a recommended list of filters, and a comment on each filter. I will deliberately tailor my recommendation as a "good place to start". Once you see how the spam you receive affects you, you can then tailor your filter order to your needs.

One Possible and Probably Good List of Filter Priorities

Friends List
This is also known as a whitelist. Add the email address of the friends, family, or business contacts who contact you regularly. Don't bother doing this all in one sitting. If you find email from friends in the training/recycle area, then right click on the email, and click "add sender to my friends".

Note the following:
1) Anyone added to the Friends List will bypass 100% of your filters, and messages from them won't show up in your training area (however, by default, such messages will be auto trained on). Their emails will go straight through to your client - that's the whole idea.
2) You may have dear friends who send you normal mail, but forward you on jokes/recipes/whatever that they also forward onto 26 other people. Your friends may be well meaning, but if you really don't want to read through these types of emails, then don't add that friend to your Friends List. Allow the Learning Filter to learn what you classify to be personal emails and what you consider to be "friend spam" emails.
3) Don't add wildcards to your Friends List. If you start adding things like *.aol.com, you're doomed. Any email matching anything on your Friends List gets a free ticket straight into your mail client's inbox. So only ever use full email addresses.

Blocked Senders
This is also a type of blacklist. Emails from here will bypass the rest of your filter sequence, and be classified as spam immediately. You can even have them deleted straight from your mail server without downloading them, if you so choose to configure spami like this.

Note the following:
1) Don't add normal spam to your Blocked Senders list. Why? Because most spam comes from randomly forged email addresses. So the Blocked Senders List isn't for normal spam.
2) What it IS for is all those newsletters that you used to like reading, but have since become bored with - they always come from the same sender address. Ideal for ex-friends you no longer want to talk to - they always come from the same sender address. You get the idea.

I mentioned the Friends List and Blocked Senders List in my list of filter priorities because they are invisible filters that always sit at the very top of your Filter Priorities list. We have had numerous threads in this forum asking, "where have my emails gone?" with the answer having something to to with the sender's email address being in one of these two lists. So they are real filters, and they come before all other filters by definition.

Sample Filter Priority List

(safety filters)
1. Attachment Filter
2. Hercule Filter

3. Newsletter Plugin

(safety filters continued)
4. Addressee Filter
5. Image Filter

(filters which can classify non-spam)
6. URL Filter
7. Learning Filter

(filters which can classify spam)
8. XHeader Filter
9. Spam Word Filter

(optional extras if you love being thorough)
10. Domain Filter
11. DCC Filter
12. Blacklist Filter

I'll now give you an explanation as to why I've chosen the filters in this order. That will help you learn more about my thinking process, since my selection in filters in quite deliberate.

1. Attachment Filter

This filter looks for "bad" attachments, but allows good attachments. You don't want to accept emails with bad attachments, virtually ever!

2. Hercule Filter

This filter tests for important security concerns, such as scripting, URL spoofing, and lots of other safety concerns.

Note: in my experience, you will have to disable or lessen off some of its detections.

1. Lock down the Hercule Filter so it checks for everything
2. Whenever it rejects an email that you know is valid, loosen off its requirements. If you get to '> 20', then just untick the option. So for instance, I unticked HTML [2]/more than...invalid HTML tags and loosened off a couple others

3. Newsletter Filter

Since newsletters might not correctly have you in the to: address field or have other complications, this filter is to accept emails that would normally be rejected by later filters.

Warning: Do NOT add newsletters to this filter until they've previously passed the Hercule filter. Many newsletters contain links and imbedded images which do "report home" that you've opened the email. You may not want this - I certainly delete all newsletters that use such evil tricks. So only add newsletters to this filter if your 100% they behave.

4. Addressee Filter

This filter, alone, catches a huge % of spam mail for me.

First, a bit of background. When emails are sent, there are two sets of addresses - identical, in a way, to how normal mail is written.

Firstly, there is the envelope. It has a to and a from sender on it. This is the same as a normal letter, right? The envelope's "to address", called the RCPT-TO address, will always be your real email address. Otherwise you won't receive it. The MAIL-FROM can be any old envelope from address the sender makes up, and can be completely ficticious.

Note: unlike regular mail, this envelope is thrown away before you collect the email from your mail server, so you WON'T get to see this.

There are another set of addresses inside the mail. These are the To: and From: addresses contained in the header of the email. This is the to and from addresses you're familiar with.

The From: address is equivalent to the letter head or the from address people place up at the top of business letters. This can be entirely made up, and doesn't have to match the Envelope MAIL-FROM address.

The To: address is equivalent to the "Dear <your name>" salutation on a normal letter. This can be entirely made up, and doesn't have to match the Envelope RCPT-TO address. This is why you receive mail that appears not addressed to you. The Envelope RCPT-TO address was addressed to you, but the To: address inside the envelope is not actually used to deliver mail, and so can be completely made up.

The Addressee Filter is brilliant in that it will screen out 100% of emails which have faked your to: address. It handles correctly emails where you are in the CC: field, and even handles correctly mail where the sender placed you in their BCC: field (according to my tests).

5. Image Filter

This filter doesn't reject emails containing images or image attachments, as the name might suggest, but rather emails containing imbedded links to images that are stored on the web. However, these images would normally be fetched from the web and appear in your email as if they were actually there all along.

This behaviour in a classic sign of spammers wanting to detect that you have opened their email (and hence accessed their server to fetch the images). Image URLs can easily have your email address embedded in them. "Please fetch the image of my logo, oh, and by the way, bob@somewhere.com is the one who is asking for this image". Warning!

The only time when emails have the right to access external images, are in newsletters where this technique is used to cut down the size of the email. Hence, we placed this filter after the Newsletter Plugin. But beware, some newsletters try to track you, anyway. Don't be adding them to your newsletter filter.

Filters which can classify non-spam

Now we can relax a bit. We've filtered out the "dangerous" spam, and we now just have to deal with the annoying spam.

I've placed the filters that can identify non-spam (or both) before the filters that can identify spam only. Why? Because false positives, cases where your good email is classified as spam, is evil, nasty, and unasseptible (to quote Supernanny). So we give the non-spam detecting filters the first go at your emails. If they do accidently let through a spam - a false negative - then it's not really a crime (only a misdemeanor)

The URL and Learning Filter both learn (everytime you train on messages), and will automatically adapt to what you consider to be good and bad mail. Brilliant.

Filters which can classify spam

And that just leaves the spam-only filters.

8. XHeader Filter

This filter runs checks in the mail header. For some of you, this filter might seem to an unusual choice, but remember, the Learning Filter only checks email contents. So to some extent, the email header has got away scott free.

I include this filter because, for me, it detects more mail than nearly any other filter! But you must configure it right.

A while back, I noticed some really simple header filtering strings that seem to screen out a huge percentage of spam. So you need to configure them into the filter.

I use the following rules in the XHeader Filter:
. Spam if 'X-Mailer' contains 'SquirrelMail'
. Spam if 'X-Spam-Score' is more than 4
. Spam if 'Message-Id' doesn't contain '@'

The one that makes the most intuitive sense is the X-Spam-Score field. Your using the spam score given by a product called SpamAssassin. This is a server based spam filter which many email servers use. So let them do all the hard work, and you just tap into their results. Adjust the value (in my case 4) as low as possible, but yet high enough to not trap any good mail.

The other rules I've found come just from analysing email headers and noticing trends. If any of these rules produce false positives, drop them.

Optional extras if you love being thorough

I've found some false positives with some of these filters, but I still think they have lots of potential.

12. Blacklist Filter

If an email that you know is OK is triggered by a blacklist, drop it! They are notoriously inaccurate, and contain wide ranges of IP addresses. Most of my personal email addresses have been flagged by at least one blacklist. So be prepared to drop any list that gives you a false positive.


There are many other plugins for Spami created by many talented programmers. I've included the plugs that protect your safety or that, in my experience, trap a good percentage of spam. Note, that several plugins overlap in the areas which they cover, so I've only included one from each area.

That's it. That, I believe, is a reaonable order and place to start for spami users who aren't ready to make their own choices about plugin order. I've also explained, extensively, why I've chosen that order, so that will give you some insight into the decision process of ordering filters.

[Gast]

Hi Zaxon

Thanks for that very informative and helpful as normal

I'd just like to say (I know I'm probabaly talking rubbish but it works for me! ) I just couldn't understand the default filter settings..

'If a filter finds a NON spam mail stop filtering'
'If a filter finds a SPAM mail stop filtering'

Either way it's going to stop filtering (in my 'eyes') and never go on to the next one...

Even though I don't get huge amounts of spam I was gettings loads through every day so I changed the default settings to...

'If a filter finds NON spam continue to next filter'

'Spami' now works SUPERBLY 100% SUCCESFULL so far since I altered the settings, I don't care if that's wrong it works for me on my PC

I love Spamihilator

All the best & thanks for the thorough info.
Andy

[Zaxon]

Thanks for that very informative and helpful as normal

Well, it keeps me off the streets.

I'd just like to say (I know I'm probabaly talking rubbish but it works for me! ) I just couldn't understand the default filter settings..

'If a filter finds a NON spam mail stop filtering'
'If a filter finds a SPAM mail stop filtering'

Either way it's going to stop filtering (in my 'eyes') and never go on to the next one...

Even though I don't get huge amounts of spam I was gettings loads through every day so I changed the default settings to...

'If a filter finds NON spam continue to next filter'

'Spami' now works SUPERBLY 100% SUCCESFULL so far since I altered the settings, I don't care if that's wrong it works for me on my PC

Hey - who can argue with success, right?

But for all those other people who might be reading that want to know about those settings, and don't have the natural success which Andy has...

Each Spami filter has three outcomes when processing an email:

1) The email is spam
2) The email is not spam
3) I'm not sure

Only two options appear under the behavior configurations, which is why some people get confused.

Spam only filters, such as the Server Filter, Blacklist filter, etc, either detect spam (option 1) or they're not sure (option 3). They don't ever trigger option 2.

So take the blacklist filter, for example. If the IP addresses in the email header match a blacklist, then it triggers it as spam. Makes sense so far. But if it doesn't find a match, it doesn't mean that it's not spam. It's just it's not sure.

In contrast, some filters, such as the whitelist filter, only trigger for not spam (option 2) or else they're not sure (option 3). They never trigger option 1. That's not their job.

The URL and Learning filter have the ability to return any of the three states. Aren't they clever?

Therefore, the correct way of having those settings is the default:

1) If you're a filter that detects spam, and you do find some, then stop all processing.
2) If you're a filter than detects non-spam, and you find that an email is definitely not spam, then stop processing.
3) If you're any sort of filter, and you're not sure a message is definitely spam or not spam, then fall on through to the next filter.

I know it might seem a little bit confusing at first, because there are only two options available for configuration. But remember, the "not sure so continue with the next filter" is there by default.

The other thing, Spami is not used as a heuristic filter, where each filter adds a probability and then someone tallys up at the end. The way it's used, at the moment, is that most filters say 100% yes or 100% no (with some exceptions). So because of this, you mostly don't want messages which have been already marked as spam, to continue going through other filters.

[Gast]

Ahhhh I SEE

All clever stuff, as I said before (sory to be boring) if there was some proper decent help files that explained how things work there wouldn't be the need to keep posting on the forum.

Just a suggestion but your posts are so informative and helpful is there any way they could be 'set up' at the top of the forum as READ ONLY posts for the important matters, such as the filter priority settings, UNDERSTANDING the default settings etc.

I'm sure it would be VERY helpful to alot of people and save your time keep going over the same Main topics again and again or worse still your 'hard work' dissapearing into Forgotton Forum history !

If you can't do that yourself and Michel reads these Forums then YOU do it please, great program but do yourself a favour and let people understand how it works PROPERLY.You made it so you know but we aren't all PC experts

All the best
Andy

[Zaxon]

Ahhhh I SEE

All clever stuff, as I said before (sory to be boring) if there was some proper decent help files that explained how things work there wouldn't be the need to keep posting on the forum.

Just a suggestion but your posts are so informative and helpful is there any way they could be 'set up' at the top of the forum as READ ONLY posts for the important matters, such as the filter priority settings, UNDERSTANDING the default settings etc.

I'm sure it would be VERY helpful to alot of people and save your time keep going over the same Main topics again and again or worse still your 'hard work' dissapearing into Forgotton Forum history !

If you can't do that yourself and Michel reads these Forums then YOU do it please, great program but do yourself a favour and let people understand how it works PROPERLY.You made it so you know but we aren't all PC experts

Michel is the one to see about making posts into stickies, as he holds that ability.

I would agree. There is some great information around that spami users would definitely benefit from being kept readily accessible all in one place.

[Bob Loeffler]

Hi Zaxon and Andy,

This topic is now a "sticky" note.

Bob

[Zaxon]

This topic is now a "sticky" note. Bob

Ah, the excellent service we could only expect from Bob Loeffler. Thanks Bob.

[Bob Loeffler]

HAHAHA!!!! You are a funny person, Zaxon!

Bob

[Gast]

Thanks Zaxon and Bob

Having put my request in for a filter priority list I then uninstalled Spami and decided to use the spam filter in Thunderbird.

However, having seen your kind, informative responses I am now back with Spami.

Ian

[Zaxon]

Having put my request in for a filter priority list I then uninstalled Spami and decided to use the spam filter in Thunderbird.

However, having seen your kind, informative responses I am now back with Spami.

A "hit and run" forum poster, hey? Well, good to see you back.

[michel]

God job! Very well done. Thanks!

Sincerely,
Michel Krämer

[Zaxon]

God job! Very well done.

Thanks Michel. It's rewarding to write about the program into which you and the plugin coders have put so much excellent work.

[Rein]

Hallo Zaxon!

However, I can also appreciate that there are lots of users of spami who are new, and so have no history to draw on. And I can appreciate that there are lots of users who don't understand, as yet, exactly what all the filters do. And so, a recommended list of filter priorities could be useful.

Is it possible to place an update? There are since 2005 a lot of new filters which are recommended. It would be greatly appreciated!

Gruß
Rein