Rule Filter Information

[boat_broker]

I have Spami 9.9.53 and am trying to use the Rule Filter to delete any email with a blank "Re" or "Fwd" subject line. Is there a symbol or command for Spami to recognise a "blank" or an empty space? I would want it to pass along an email that had the subject "Re: Words" but to mark "Re: " as spam. I woud expect that the blank area could be one or more characters that were "invisable", though I suspect that they are just a blank field.

I checked the Wiki and noticed that there were no instructions for the Rule Filter. Is there any other resource to check?

Thanks
Mike

[Chactory]

Hi Mike!

Perhaps you can define your demanded rule with a regular expression. Please read about regexes in the online help.

Regards, Chactory

[boat_broker]

Thanks Chactory,

I had considered using Regular Expressions before but I was too lazy to try and learn how they worked. When I saw the new “Rules” Section I was kind of hoping that it might do the same thing easier. However, I have yet to see any update on just how to use the Rules Section. Is there any discussion of that section or instructions for that section?

Getting back to regular expressions. After playing around with them for a while I believe that the RegEx \b[Rr][Ee]\b:\s\B will do what I was asking, to knock out any email with a blank after the “Re:” while not eliminating a message with a word after the blank. I extrapolated this to Fw: and Fwd: as well (\b[Ff][Ww]\b:\s\B and \b[Rr][Ee]\b:\s\B) so I believe that I am covered. Comments would be welcome.

It looks like the Rules Section may allow me to use RegExs in specific areas of an email (ie Subject or body, etc). I am looking forward to exploring the potential of this "wizard".

While this note was responding to Chactory, any responses would be welcome.

Mike

[Quellcore]

I had considered using Regular Expressions before but I was too lazy to try and learn how they worked. When I saw the new “Rules” Section I was kind of hoping that it might do the same thing easier. However, I have yet to see any update on just how to use the Rules Section. Is there any discussion of that section or instructions for that section?

We are all pretty excited about the Rule Filter, it is very powerful and could even substitute several other filters if you really wanted to put the extra effort into it.
The usage is pretty straight forward, in fact there is a german WIKI-Entry which hasn't been translated to English yet. The only tricky part is to manhandle Regular Expression, which is a whole topic itself. People have written Books about it, you know your way around in that language and creating rules for the Rules Filter containing Regular Expression will be quick and painless.
I'm making extensive use of the Rulefilter myself, although extensive in the meaning of quantity, i am also trying hard to improve my knowledge about Regular Expressions and incorporate that into new Rules on a daily basis. I will introduce my most effective Rules here in this board in the very near future, but it might be in the german section. Other Board memebrs and myself tried to come up with strategy how to discuss Rules for the Rulefilter. Stuff like what to filter, what not to filter, what to watch out for in terms of false positives etc ...
We started discussing it, but obviously there is no "Project" startet yet

Getting back to regular expressions. After playing around with them for a while I believe that the RegEx \b[Rr][Ee]\b:\s\B will do what I was asking, to knock out any email with a blank after the “Re:” while not eliminating a message with a word after the blank. I extrapolated this to Fw: and Fwd: as well (\b[Ff][Ww]\b:\s\B and \b[Rr][Ee]\b:\s\B) so I believe that I am covered. Comments would be welcome.

Seeing you experimenting with special features like word boundaries makes me happy, it is generally a good way of eliminating fasle positives.

Ok lets strip it down:
Your original RegExp:

Code: Alles auswählen

\b[Rr][Ee]\b:\s\B

1. Regex Matching Mode is "case insensitive"
   Since Spami uses the "case insensitive" flag in the used RegExp-Libray, it means the Rulefilter and the Spamword Filter are case insensitive.
   Therefor your rule could already be simplified to
   

   Code: Alles auswählen

   \bRE\b:\s\B

2. Word Boundaries "\b"
   I'm still having trouble grasping the effect of word boundaries in special and advanced scenarios.
   

   1. Before the first character in the string, if the first character is a word character.
   2. After the last character in the string, if the last character is a word character.
   3. Between two characters in the string, where one is a word character and the other is not a word character.

   
   Your first "\b" before "RE" makes sense.
   It covers Scenario "A". The only problem is that it could also cover Scenario "C".
   A Word Character is covered be the class [0-9a-z] while a Non-Word Character is everything else, so your RegExp would also find a match in the String "BA.RE:"
   To avoid this i would recommend looking into the use of anchors for the start and end of the line instead of using word boundaries.
   Your second "\b" after "E" and before ":" doesn't make any sense. Between "E" and ":" there is always a word boundary by definition "C" (see above) so you could leave it out.
3. Negated Word Boundaries "\B"
   I can't figure out your intention in this case

As for matching a RegExp, the Rule filter offers two different options, "contains (regex)" and "matches (regex)".
The easiest and shortest solution to your task i could come up with this one:

Code: Alles auswählen

Subject matches (regex) RE:\s*

Or as an example hot to utilize anchors:

Code: Alles auswählen

Subject contains (regex) ^RE:\s*$

It looks like the Rules Section may allow me to use RegExps in specific areas of an email (ie Subject or body, etc). I am looking forward to exploring the potential of this "wizard".

The same for me and some others here in this forum.
Non of us is really a pro when it comes to RegExps, we simply like to explore and experiment with the endless possibilities.
RegEx implementations come in many differernt flavors like Perl, Java and .Net, they have slight differences, so experimenting is sometimes the only way of telling if something works or not.
Spamihilator is using the Boost RegEx Library which is a PERL flavor.
Check out the 4 different matching modes.
There is no official statement which one is turned within Spamihilator.
It's pretty obvious that "case insensitive" matching is enabled, about the rest i don't know.

Regards,
Quellcore

[Chactory]

Hi boat_broker and Quellcore!

This discussion is becoming more and more excellent!

Hmm ... how I understand it, Mike was searching for a RegEx, which will recognize a subject line containing the "RE:" passus only. So he seems to have searched for something like a negation with a RegEx, sort of "nothing more follows". That's why he was trying out the "\B", which meant to recognize that no word follows. Until now, I have no idea how to describe with a RegEx, that the subject contains "RE:", and nothing more ...

Cheers, Chactory

[Quellcore]

Until now, I have no idea how to describe with a RegEx, that the subject contains "RE:", and nothing more ...

I provided 2 working solutions that i verified in my Virtual Machine, my lengthly post might have distracted you from the solutions:

Best and shortest:
Subject matches (regex) RE:\s*

Rule.png (10.95 KiB) 1867-mal betrachtet

Or as an example hot to utilize anchors:
Subject contains (regex) ^RE:\s*$

Make sure to realize the difference between the most common "contains (regex)" and the "matches (regex)"

Please also note that Mike's proposal \b[Rr][Ee]\b:\s\B doesn't work the way he would have liked, i tried it in the VM and it didn't work in any of the cases.

Regards,
Quellcore

[Chactory]

Hi Quellcore,

thank you for explaining! Now I got it!

realize the difference between the most common "contains (regex)" and the "matches (regex)"

Regards, Chactory

[boat_broker]

Quellcore,

Thanks for your expansive reply. I am still trying to digest it.

I believe that you correctly identified my intent with \b[Rr][Ee]\b:\s\B. However, with the RegEx Coach VM it does seem to work. It recognizes "Re:" and "Re: " and "Re: " (a re followed by a tab). At the same time it passes "Re: boats" and "Re:boats". So far, all is good.

I will let you know if I change anything and I will go back to my formula and try to explain my reasoning. I do need to review the "anchor" concept that you mentioned. I should also note that I came up with this as an entry into the Spam Words file not the Rule Filter. I believe that it will be the same in the Rule Filter but it should be a little more precise.

Thank you so much for the time you put into your reply. It was way more than I had hoped.

Mike

[boat_broker]

Quellcore,

I’ve been pecking away at this today and my head is spinning. I am curious if the virtual machine that you are using is available. I have been getting different results than you mention (or I am interpreting them incorrectly) when I use The Regex Coach by Edi Weitz that is mentioned in the Spamihilator Wiki RegEx Tutorial

Thanks,
Mike

[Quellcore]

I believe that you correctly identified my intent with \b[Rr][Ee]\b:\s\B. However, with the RegEx Coach VM it does seem to work. It recognizes "Re:" and "Re: " and "Re: " (a re followed by a tab). At the same time it passes "Re: boats" and "Re:boats". So far, all is good. I will let you know if I change anything and I will go back to my formula and try to explain my reasoning. I do need to review the "anchor" concept that you mentioned. I should also note that I came up with this as an entry into the Spam Words file not the Rule Filter. I believe that it will be the same in the Rule Filter but it should be a little more precise.

I finally got your RegExp to work in The RegEx Coach, however even then it would also match a subject like "BA RE:". This might be alright for you, i just wanted to point it out that there could be any text before that "RE:" as long as its separated from "RE:" by a word boundary (Non-Word-Character).
But when i tried your RegExp in Spami it would NOT work in any of the cases you see in my screenshot of the Training Area above.
I understand your reasoning behind your RegEx, now even the "\B" at the end
Except for the problem with text before the "RE:" it's fine, i just couldn't get it to work in Spami.
I cannot tell you why it is not working but it must have to do with the word boundaries.

Thank you so much for the time you put into your reply. It was way more than I had hoped.

I'm also still trying to learn more about RegExps myself, i'm sure i'll benefit from this discussion myself, so call me selfish if you want .

I’ve been pecking away at this today and my head is spinning. I am curious if the virtual machine that you are using is available. I have been getting different results than you mention (or I am interpreting them incorrectly) when I use The Regex Coach by Edi Weitz that is mentioned in the Spamihilator Wiki RegEx Tutorial

I'm still using "The RegEx Coach" myself from time to time, but unfortunately it doesn't seem to have the exact same interpretation of PERL RegExps as being used in Spami through the Boost Library.
Using this fantastic program for developing RegExps for single Words should be no problem, i have done that myself plenty of times with great success.
But when it comes to more complex expressions with word boundaries, anchors and linefeeds i don't trust "The RegEx Coach" anymore since i already had more than one incident where it worked in the program but not in Spami.
One example:
When you have a text consisting of two lines of text with a forced linebreak (Hit "ENTER" at the end of the line") in between those two lines you can match this linebreak with the RegEx "\s" in "The RegEx Coach" but in the Boost Library it's two times "\s".
Every RegExp Program out there looks for a substring match of your RegExp. This corresponds to the method "contains (regex)" while Spami offers also the method "matches (regex)" where the whole test-text has to be matched by your RegExp. This makes it a little harder to develop RegExps that uses the method "matches (regex)".
My newest favorite playground for RegExps is the QuickREx plug-in in development platform Eclipse.
When I was trying to find the URL i just learned that QuickREx is also available now a s a standalone application.
http://www.bastian-bergerhoff.com/eclipse/features/web/QuickREx/standalone.html
The behavior is way closer to the one in Spami.
Of course the problem with the different methods "contains (regex)" and "matches (regex) persists.
Choose the Jakarta ORO Perl implementation and set the global ORO Perl flag "Case insensitive"
This requires only checking/unschecking stuff, don't panic

Regards,
Quellcore

[boat_broker]

Thanks Quellcore,

I downloaded the QuickREx program that you linked to and went to set it up. However, in the Global Flags section I found the check box for the "ORO Perl Flags (Case insensitive)" but not for "Jakarta ORO Perl". There is one for "Jakarta-Regexp Flags (Case insensitive) (Multiline)", is that what you were referring to?

Out of curiosity, how did you determine to select those two implementations?

Obviously, I'm going to have to send myself some emails. My RegEx seemed to be working in my Spami but maybe it was focused on something else. Like you mentioned, the two different RegEx engines give me different results.

Mike

[Quellcore]

I downloaded the QuickREx program that you linked to and went to set it up. However, in the Global Flags section I found the check box for the "ORO Perl Flags (Case insensitive)" but not for "Jakarta ORO Perl". There is one for "Jakarta-Regexp Flags (Case insensitive) (Multiline)", is that what you were referring to?

Sorry for the confusion, i was still using the Plug-in in Eclipse which looks almost identical but not quite.

Evaluating: ORO Perl 5
Global Flags: ORO Perl "Case Insensitive"

Since you chose "ORO Perl 5" Evaluation only the "ORO Perl" Flags matter, all the others don't have any effect as long as "ORO Perl 5" Evaluation is selected.

Depending on what kind of RegEx i want to create i also enable "Multiline Mode" (ORO Perl of course), this is only important when you want to make use of the anchors "^" (Start of Line) and "$" (End of Line).
The Multiline Mode was perfect for your project, the empty subject with "RE:".
This way i was able to check all possinilities at the same time:

QuickREx_01.png (5.01 KiB) 1825-mal betrachtet

Out of curiosity, how did you determine to select those two implementations?

I know Spami uses the boost.regex library which is documented very well.
Apparently you could compile boost.regex for three different regular expression syntax (so called "flavors" i believe), PERL, POSIX extended and POSIX Basic.
Now i looked at the Regular Expression Flavor Comparison Chart.
There a plenty of features that DO work in Spami that indicate it has to be PERL and cannot be POSIX.
I have done a lot of RegExps in Spami which make it very clear that the boost.regex was compiled with "Case Insensitive" enabled.

Obviously, I'm going to have to send myself some emails. My RegEx seemed to be working in my Spami but maybe it was focused on something else. Like you mentioned, the two different RegEx engines give me different results.

There is a pretty quick and convenient way for testing Rules in Spamihilator:

• Get Boris Kugelmeier's Virtual Pop3 Server (not developed anymore, but works and its free) <here>
• Start the Program and adjust the Port if you have to
• Click the "Anleitung" Button, you will find instruction even in english how set up an account in you mailclient that would connect to the virtual server through Spami and how to get the Testmails on the Server.
• Click "Hochfahren" to turn the virtual server on and check mails for your virtual account.

Regards,
Quellcore

[Chactory]

Hi Quellcore,

Evaluating: ORO Perl 5
Global Flags: ORO Perl "Case Insensitive"

where can I find the Global Flags settings? I didn't success to find it since two days ...

Sincerely, Chactory

[Quellcore]

Hallo Chactory!

at the very bottom, you might have to unfold "Global Flags"

Gruß
Quellcore

[Quellcore]

where can I find the Global Flags settings? I didn't success to find it since two days ...

You can also see it in my last screenshot, btw ...

@ Mike and Chactory:
Did you play with QuickREx yet? How do you like it, could you get used to it and disregard The Regex Coach?


Regards,
Quellcore