Sequence of Plugin Filters

Antwort erstellen

Bestätigungscode
Gib den Code genau so ein, wie du ihn siehst; Groß- und Kleinschreibung wird nicht unterschieden.
Smilies
:D :) :( :o :shock: :? 8) :lol: :x :P :oops: :cry: :evil: :twisted: :roll: :wink: :!: :?: :idea: :arrow: :| :mrgreen:
BBCode ist eingeschaltet
[img] ist eingeschaltet
[flash] ist ausgeschaltet
[url] ist eingeschaltet
Smilies sind eingeschaltet
Die letzten Beiträge des Themas
   

Ansicht erweitern Die letzten Beiträge des Themas: Sequence of Plugin Filters

Re: update list of priorities

Beitrag von Chactory » 21. Dez 2008, 14:57

Hello Rein,

thank you for your contribution. Since Zaxon didn't answer until now, I have looked over his postings and updated some sentences. I hope that this could be useful for you.

Zaxons recommendations updated (concisely)

Filter sequence

1. Filters which detect NON-SPAM only should be set to the highest/first position.
2. Filters that protect your safety come second.
3. Strong and accurate filters which detect SPAM only follow now, still at high priority.
4. Special case filters which detect SPAM only should follow in the middle field of the sequence.
5. Spam Word Filter detecting SPAM only and Learning Filter classifying SPAM and NON-SPAM come at last.
6. Configure your filters following your own needs and habits.
7. Watch your results using the Statistics Plugin, and promote accurate filters up the list, but place less accurate filters down the list or delete it.
8. Care for FALSE-POSITIVES much more than for FALSE-NEGATIVES.
9. Use the friends list carefully, but use it, particularly for important business partners and your chef.
10. Don't pollute the blocked senders list with all the SPAM you get, only add repeating SPAM senders and unwanted newsletters.
11. Be careful with wildcards in the lists, better don't add any.
12. Watch your newsletters, because they may track you with links or imbedded images with links, and they may pollute your Learning Filter.
13. With the expert options, you may configure the filter behaviour of every filter, but act carefully, again.

Recommended Filters and Extras

- distributed within the Spamihilator installer:

· Friends List 1) allows mails from senders contained in the list.
· Blocked Senders blocks mails from senders listed.

· Attachment Filter blocks mails with defined attachments.
· DCC Filter filters mass mails recognized by a hash count delivered to the DCC extracted from mails of the DCC contributers.
· Image Filter filters mails containing images reloading from the web.
· Learning Filter (Bayesian Filter) allows or filters mails by assessing their good words and spam words.
· Link Filter filters mails containg spam urls.
· Newsletter Filter filters mails with a listserver adress or a subject entered in it's list.
· Spam Word Filter sieves mails containing pre-defined spam words. (You can download an updated << spampoint.xml >> file and copy it into the programs\spamihilator folder, just overwrite the older file.)

- distributed as plugin:

· Addressee Filter 2) filters mails not addressed to one of your mail addresses.
· AIR Filter 0.1.5 2) filters mails whose recipients are suspiciously similar.
· Attachment Extensions Filter v0.9.6 2) sieves or allows mails with specific attachments.
· Charset Plugin 3) recognizes mails with unusual character sets.
· Empty Mail Filter v1.3.2 filters mails which are empty or only containing an inline attachment or an attachment.
· Foreign language Filter 2) recognizes spam mails in some foreign languages.
· Hercule Filter 4) sieves mails showing some typical spam behaviour.
· Misnamed Filter compares the mail address with the fitting real name.
· Mystic-Signs-Filter 1.1.1 2) filters mails containing weird characters in the subject line.
· Outgoing Friends keeps the addresses you send mails to as friends.
· Scripts Filter v1.2.2 recognizes mails containing scripts.
· Signature Filter 0.5.2 2) allows mails when containing a defined signature.
· Strange-Country-Filter v 1.1.0 2) filters mails coming from strange countries.
· Trust SpamAssassin Headers allows mails containing SpamAssassin appraisal.
· Unclassified Filter v1.2.2 sorts out all mails not classified by Spamihilator.

· LiveStats allows to show your Spamihilator statistics in your signature.
· Filter Statistics v1.1.3 shows a comprehensive statistic analysis.

1) My favorites printed in bold fonts.
2) Not updated, but still working with me. If no installer, perhaps you have to copy the dll file into the program plugin folder.
3) Not yet updated, still working, copy the dll file into the program plugin folder. I'm using a newer inofficial beta.
4) Very promising, but not yet updated, false positive filterings when not set accurately.

Regards, Chactory

Re: update list of priorities

Beitrag von Rein » 13. Dez 2008, 18:29

Hallo Zaxon!
However, I can also appreciate that there are lots of users of spami who are new, and so have no history to draw on. And I can appreciate that there are lots of users who don't understand, as yet, exactly what all the filters do. And so, a recommended list of filter priorities could be useful.


Is it possible to place an update? There are since 2005 a lot of new filters which are recommended. It would be greatly appreciated!

Gruß
Rein

Beitrag von Zaxon » 17. Jun 2005, 09:29

michel hat geschrieben:God job! Very well done.

Thanks Michel. It's rewarding to write about the program into which you and the plugin coders have put so much excellent work.

Beitrag von michel » 16. Jun 2005, 20:00

God job! Very well done. Thanks!

Sincerely,
Michel Krämer

Re: Thanks Zaxon and Bob

Beitrag von Zaxon » 16. Jun 2005, 10:16

Ian hat geschrieben:Having put my request in for a filter priority list I then uninstalled Spami and decided to use the spam filter in Thunderbird.

However, having seen your kind, informative responses I am now back with Spami.

A "hit and run" forum poster, hey? Well, good to see you back.

Thanks Zaxon and Bob

Beitrag von Gast » 16. Jun 2005, 05:42

Thanks Zaxon and Bob

Having put my request in for a filter priority list I then uninstalled Spami and decided to use the spam filter in Thunderbird.

However, having seen your kind, informative responses I am now back with Spami.

Ian

Beitrag von Bob Loeffler » 15. Jun 2005, 20:14

HAHAHA!!!! You are a funny person, Zaxon! ;-)

Bob

Beitrag von Zaxon » 15. Jun 2005, 20:11

Bob Loeffler hat geschrieben:This topic is now a "sticky" note. :-)Bob

Ah, the excellent service we could only expect from Bob Loeffler. Thanks Bob.

Beitrag von Bob Loeffler » 15. Jun 2005, 19:15

Hi Zaxon and Andy,

This topic is now a "sticky" note. :-)

Bob

Beitrag von Zaxon » 15. Jun 2005, 17:31

Andy hat geschrieben:Ahhhh I SEE :D

All clever stuff, as I said before (sory to be boring) if there was some proper decent help files that explained how things work there wouldn't be the need to keep posting on the forum.

Just a suggestion but your posts are so informative and helpful is there any way they could be 'set up' at the top of the forum as READ ONLY posts for the important matters, such as the filter priority settings, UNDERSTANDING the default settings :wink: etc.

I'm sure it would be VERY helpful to alot of people and save your time keep going over the same Main topics again and again or worse still your 'hard work' dissapearing into Forgotton Forum history !

If you can't do that yourself and Michel reads these Forums then YOU do it please, great program but do yourself a favour and let people understand how it works PROPERLY.You made it so you know but we aren't all PC experts :?

Michel is the one to see about making posts into stickies, as he holds that ability.

I would agree. There is some great information around that spami users would definitely benefit from being kept readily accessible all in one place.

Beitrag von Gast » 15. Jun 2005, 11:44

Ahhhh I SEE :D

All clever stuff, as I said before (sory to be boring) if there was some proper decent help files that explained how things work there wouldn't be the need to keep posting on the forum.

Just a suggestion but your posts are so informative and helpful is there any way they could be 'set up' at the top of the forum as READ ONLY posts for the important matters, such as the filter priority settings, UNDERSTANDING the default settings :wink: etc.

I'm sure it would be VERY helpful to alot of people and save your time keep going over the same Main topics again and again or worse still your 'hard work' dissapearing into Forgotton Forum history !

If you can't do that yourself and Michel reads these Forums then YOU do it please, great program but do yourself a favour and let people understand how it works PROPERLY.You made it so you know but we aren't all PC experts :?

All the best
Andy

Beitrag von Zaxon » 14. Jun 2005, 15:52

Andy hat geschrieben:Thanks for that very informative and helpful as normal :D

Well, it keeps me off the streets.

Andy hat geschrieben:I'd just like to say (I know I'm probabaly talking rubbish but it works for me! ) I just couldn't understand the default filter settings..

'If a filter finds a NON spam mail stop filtering'
'If a filter finds a SPAM mail stop filtering'

Either way it's going to stop filtering (in my 'eyes') and never go on to the next one...

Even though I don't get huge amounts of spam I was gettings loads through every day so I changed the default settings to...

'If a filter finds NON spam continue to next filter'

'Spami' now works SUPERBLY :D 100% SUCCESFULL so far since I altered the settings, I don't care if that's wrong it works for me on my PC
:D :D :D :D :D :D :D :D

Hey - who can argue with success, right?

But for all those other people who might be reading that want to know about those settings, and don't have the natural success which Andy has...

Each Spami filter has three outcomes when processing an email:

1) The email is spam
2) The email is not spam
3) I'm not sure

Only two options appear under the behavior configurations, which is why some people get confused.

Spam only filters, such as the Server Filter, Blacklist filter, etc, either detect spam (option 1) or they're not sure (option 3). They don't ever trigger option 2.

So take the blacklist filter, for example. If the IP addresses in the email header match a blacklist, then it triggers it as spam. Makes sense so far. But if it doesn't find a match, it doesn't mean that it's not spam. It's just it's not sure.

In contrast, some filters, such as the whitelist filter, only trigger for not spam (option 2) or else they're not sure (option 3). They never trigger option 1. That's not their job.

The URL and Learning filter have the ability to return any of the three states. Aren't they clever?

Therefore, the correct way of having those settings is the default:

1) If you're a filter that detects spam, and you do find some, then stop all processing.
2) If you're a filter than detects non-spam, and you find that an email is definitely not spam, then stop processing.
3) If you're any sort of filter, and you're not sure a message is definitely spam or not spam, then fall on through to the next filter.

I know it might seem a little bit confusing at first, because there are only two options available for configuration. But remember, the "not sure so continue with the next filter" is there by default.

The other thing, Spami is not used as a heuristic filter, where each filter adds a probability and then someone tallys up at the end. The way it's used, at the moment, is that most filters say 100% yes or 100% no (with some exceptions). So because of this, you mostly don't want messages which have been already marked as spam, to continue going through other filters.

Beitrag von Gast » 14. Jun 2005, 11:59

Hi Zaxon

Thanks for that very informative and helpful as normal :D

I'd just like to say (I know I'm probabaly talking rubbish but it works for me! ) I just couldn't understand the default filter settings..

'If a filter finds a NON spam mail stop filtering'
'If a filter finds a SPAM mail stop filtering'

Either way it's going to stop filtering (in my 'eyes') and never go on to the next one...

Even though I don't get huge amounts of spam I was gettings loads through every day so I changed the default settings to...

'If a filter finds NON spam continue to next filter'

'Spami' now works SUPERBLY :D 100% SUCCESFULL so far since I altered the settings, I don't care if that's wrong it works for me on my PC
:D :D :D :D :D :D :D :D
I love Spamihilator :wink:

All the best & thanks for the thorough info.
Andy

Beitrag von Zaxon » 14. Jun 2005, 10:32

Ian hat geschrieben:Thanks for that.

Could I ask you to list your filters in order.

I'm willing to bet there are a lot of people like me that want a standard product to highlight spam. I'm not too worried about getting that extra 1% accuracy at this moment in time - I'm happy to kill spam: full stop.

I need somebody to say I use these plug-ins in this order and it works for me. As time progresses and I get used to the programme I will make changes myself. For instance, how do I know which filters work the best??

Spami has a great reputation. However, as a new user to be offered a big choice of filters and not being sure what to do probably loses Spami some users. It's nice to have choice when you know what you are doing, otherwise, you go with a product that can't be customised and make do.

The spirit of this thread was to give people a way of looking at their own spam history, and giving them guidelines on how to tailor their filter order to their changing circumstances.

However, I can also appreciate that there are lots of users of spami who are new, and so have no history to draw on. And I can appreciate that there are lots of users who don't understand, as yet, exactly what all the filters do. And so, a recommended list of filter priorities could be useful.

So, with that in mind, I'll provide you a recommended list of filters, and a comment on each filter. I will deliberately tailor my recommendation as a "good place to start". Once you see how the spam you receive affects you, you can then tailor your filter order to your needs.

One Possible and Probably Good List of Filter Priorities

Friends List
This is also known as a whitelist. Add the email address of the friends, family, or business contacts who contact you regularly. Don't bother doing this all in one sitting. If you find email from friends in the training/recycle area, then right click on the email, and click "add sender to my friends".

Note the following:
1) Anyone added to the Friends List will bypass 100% of your filters, and messages from them won't show up in your training area (however, by default, such messages will be auto trained on). Their emails will go straight through to your client - that's the whole idea.
2) You may have dear friends who send you normal mail, but forward you on jokes/recipes/whatever that they also forward onto 26 other people. Your friends may be well meaning, but if you really don't want to read through these types of emails, then don't add that friend to your Friends List. Allow the Learning Filter to learn what you classify to be personal emails and what you consider to be "friend spam" emails.
3) Don't add wildcards to your Friends List. If you start adding things like *.aol.com, you're doomed. Any email matching anything on your Friends List gets a free ticket straight into your mail client's inbox. So only ever use full email addresses.

Blocked Senders
This is also a type of blacklist. Emails from here will bypass the rest of your filter sequence, and be classified as spam immediately. You can even have them deleted straight from your mail server without downloading them, if you so choose to configure spami like this.

Note the following:
1) Don't add normal spam to your Blocked Senders list. Why? Because most spam comes from randomly forged email addresses. So the Blocked Senders List isn't for normal spam.
2) What it IS for is all those newsletters that you used to like reading, but have since become bored with - they always come from the same sender address. Ideal for ex-friends you no longer want to talk to - they always come from the same sender address. You get the idea.

I mentioned the Friends List and Blocked Senders List in my list of filter priorities because they are invisible filters that always sit at the very top of your Filter Priorities list. We have had numerous threads in this forum asking, "where have my emails gone?" with the answer having something to to with the sender's email address being in one of these two lists. So they are real filters, and they come before all other filters by definition.

Sample Filter Priority List

(safety filters)
1. Attachment Filter
2. Hercule Filter

3. Newsletter Plugin

(safety filters continued)
4. Addressee Filter
5. Image Filter

(filters which can classify non-spam)
6. URL Filter
7. Learning Filter

(filters which can classify spam)
8. XHeader Filter
9. Spam Word Filter

(optional extras if you love being thorough)
10. Domain Filter
11. DCC Filter
12. Blacklist Filter

I'll now give you an explanation as to why I've chosen the filters in this order. That will help you learn more about my thinking process, since my selection in filters in quite deliberate.

1. Attachment Filter

This filter looks for "bad" attachments, but allows good attachments. You don't want to accept emails with bad attachments, virtually ever!

2. Hercule Filter

This filter tests for important security concerns, such as scripting, URL spoofing, and lots of other safety concerns.

Note: in my experience, you will have to disable or lessen off some of its detections.

1. Lock down the Hercule Filter so it checks for everything
2. Whenever it rejects an email that you know is valid, loosen off its requirements. If you get to '> 20', then just untick the option. So for instance, I unticked HTML [2]/more than...invalid HTML tags and loosened off a couple others

3. Newsletter Filter

Since newsletters might not correctly have you in the to: address field or have other complications, this filter is to accept emails that would normally be rejected by later filters.

Warning: Do NOT add newsletters to this filter until they've previously passed the Hercule filter. Many newsletters contain links and imbedded images which do "report home" that you've opened the email. You may not want this - I certainly delete all newsletters that use such evil tricks. So only add newsletters to this filter if your 100% they behave.

4. Addressee Filter

This filter, alone, catches a huge % of spam mail for me.

First, a bit of background. When emails are sent, there are two sets of addresses - identical, in a way, to how normal mail is written.

Firstly, there is the envelope. It has a to and a from sender on it. This is the same as a normal letter, right? The envelope's "to address", called the RCPT-TO address, will always be your real email address. Otherwise you won't receive it. The MAIL-FROM can be any old envelope from address the sender makes up, and can be completely ficticious.

Note: unlike regular mail, this envelope is thrown away before you collect the email from your mail server, so you WON'T get to see this.

There are another set of addresses inside the mail. These are the To: and From: addresses contained in the header of the email. This is the to and from addresses you're familiar with.

The From: address is equivalent to the letter head or the from address people place up at the top of business letters. This can be entirely made up, and doesn't have to match the Envelope MAIL-FROM address.

The To: address is equivalent to the "Dear <your name>" salutation on a normal letter. This can be entirely made up, and doesn't have to match the Envelope RCPT-TO address. This is why you receive mail that appears not addressed to you. The Envelope RCPT-TO address was addressed to you, but the To: address inside the envelope is not actually used to deliver mail, and so can be completely made up.

The Addressee Filter is brilliant in that it will screen out 100% of emails which have faked your to: address. It handles correctly emails where you are in the CC: field, and even handles correctly mail where the sender placed you in their BCC: field (according to my tests).

5. Image Filter

This filter doesn't reject emails containing images or image attachments, as the name might suggest, but rather emails containing imbedded links to images that are stored on the web. However, these images would normally be fetched from the web and appear in your email as if they were actually there all along.

This behaviour in a classic sign of spammers wanting to detect that you have opened their email (and hence accessed their server to fetch the images). Image URLs can easily have your email address embedded in them. "Please fetch the image of my logo, oh, and by the way, bob@somewhere.com is the one who is asking for this image". Warning!

The only time when emails have the right to access external images, are in newsletters where this technique is used to cut down the size of the email. Hence, we placed this filter after the Newsletter Plugin. But beware, some newsletters try to track you, anyway. Don't be adding them to your newsletter filter.

Filters which can classify non-spam

Now we can relax a bit. We've filtered out the "dangerous" spam, and we now just have to deal with the annoying spam.

I've placed the filters that can identify non-spam (or both) before the filters that can identify spam only. Why? Because false positives, cases where your good email is classified as spam, is evil, nasty, and unasseptible (to quote Supernanny). So we give the non-spam detecting filters the first go at your emails. If they do accidently let through a spam - a false negative - then it's not really a crime (only a misdemeanor)

The URL and Learning Filter both learn (everytime you train on messages), and will automatically adapt to what you consider to be good and bad mail. Brilliant.

Filters which can classify spam

And that just leaves the spam-only filters.

8. XHeader Filter

This filter runs checks in the mail header. For some of you, this filter might seem to an unusual choice, but remember, the Learning Filter only checks email contents. So to some extent, the email header has got away scott free.

I include this filter because, for me, it detects more mail than nearly any other filter! But you must configure it right.

A while back, I noticed some really simple header filtering strings that seem to screen out a huge percentage of spam. So you need to configure them into the filter.

I use the following rules in the XHeader Filter:
. Spam if 'X-Mailer' contains 'SquirrelMail'
. Spam if 'X-Spam-Score' is more than 4
. Spam if 'Message-Id' doesn't contain '@'

The one that makes the most intuitive sense is the X-Spam-Score field. Your using the spam score given by a product called SpamAssassin. This is a server based spam filter which many email servers use. So let them do all the hard work, and you just tap into their results. Adjust the value (in my case 4) as low as possible, but yet high enough to not trap any good mail.

The other rules I've found come just from analysing email headers and noticing trends. If any of these rules produce false positives, drop them.

Optional extras if you love being thorough

I've found some false positives with some of these filters, but I still think they have lots of potential.

12. Blacklist Filter

If an email that you know is OK is triggered by a blacklist, drop it! They are notoriously inaccurate, and contain wide ranges of IP addresses. Most of my personal email addresses have been flagged by at least one blacklist. So be prepared to drop any list that gives you a false positive.


There are many other plugins for Spami created by many talented programmers. I've included the plugs that protect your safety or that, in my experience, trap a good percentage of spam. Note, that several plugins overlap in the areas which they cover, so I've only included one from each area.

That's it. That, I believe, is a reaonable order and place to start for spami users who aren't ready to make their own choices about plugin order. I've also explained, extensively, why I've chosen that order, so that will give you some insight into the decision process of ordering filters.

Beitrag von Gast » 14. Jun 2005, 01:19

Zaxon

Thanks for that.

Could I ask you to list your filters in order.

I'm willing to bet there are a lot of people like me that want a standard product to highlight spam. I'm not too worried about getting that extra 1% accuracy at this moment in time - I'm happy to kill spam: full stop.

I need somebody to say I use these plug-ins in this order and it works for me. As time progresses and I get used to the programme I will make changes myself. For instance, how do I know which filters work the best??

Spami has a great reputation. However, as a new user to be offered a big choice of filters and not being sure what to do probably loses Spami some users. It's nice to have choice when you know what you are doing, otherwise, you go with a product that can't be customised and make do.

Thanks
Ian

Nach oben

cron

 industrious-southeast