Return-Path: guido@python.org Delivery-Date: Sun Sep 8 04:38:47 2002 From: guido@python.org (Guido van Rossum) Date: Sat, 07 Sep 2002 23:38:47 -0400 Subject: [Spambayes] test sets? In-Reply-To: Your message of "Sat, 07 Sep 2002 22:55:12 EDT." References: Message-ID: <200209080338.g883clw17223@pcp02138704pcs.reston01.va.comcast.net> > > But it also identified as spam everything in my inbox that had any > > MIME structure or HTML parts, and several messages in my saved 'zope > > geeks' list that happened to be using MIME and/or HTML. > > Do you know why? The strangest implied claim there is that it hates MIME > independent of HTML. For example, the spamprob of 'content-type:text/plain' > in that pickle is under 0.21. 'content-type:multipart/alternative' gets > 0.93, but that's not a killer clue, and one bit of good content will more > than cancel it out. I reran the experiment (with the new SpamHam1.pik, but it doesn't seem to make a difference). Here are the clues for the two spams in my inbox (in hammie.py's output format, which sorts the clues by probability; the first two numbers are the message number and overall probability; then line-folded): 66 1.00 S 'facility': 0.01; 'speaker': 0.01; 'stretch': 0.01; 'thursday': 0.01; 'young,': 0.01; 'mistakes': 0.12; 'growth': 0.85; '>content-type:text/plain': 0.85; 'please': 0.85; 'capital': 0.92; 'series': 0.92; 'subject:Don': 0.94; 'companies': 0.96; '>content-type:text/html': 0.96; 'fee': 0.96; 'money': 0.96; '8:00am': 0.99; '9:00am': 0.99; '>content-type:image/gif': 0.99; '>content-type:multipart/alternative': 0.99; 'attend': 0.99; 'companies,': 0.99; 'content-type/type:multipart/alternative': 0.99; 'content-type:multipart/related': 0.99; 'economy': 0.99; 'economy"': 0.99 This has 6 content-types as spam clues, only one of which is related to HTML, despite there being an HTML alternative (and 12 other spam clues, vs. only 6 ham clues). This was an announcement of a public event by our building owners, with a text part that was the same as the HTML (AFAICT). Its language may be spammish, but the content-type clues didn't help. (BTW, it makes me wonder about the wisdom of keeping punctuation -- 'economy' and 'economy"' to me don't seem to deserve two be counted as clues.) 76 1.00 S '(near': 0.01; 'alexandria': 0.01; 'conn': 0.01; 'from:adam': 0.01; 'from:email addr:panix': 0.01; 'poked': 0.01; 'thorugh': 0.01; 'though': 0.03; "i'm": 0.03; 'reflect': 0.05; "i've": 0.06; 'wednesday': 0.07; 'content-disposition:inline': 0.10; 'contacting': 0.93; 'sold': 0.96; 'financially': 0.98; 'prices': 0.98; 'rates': 0.99; 'discount.': 0.99; 'hotel': 0.99; 'hotels': 0.99; 'hotels.': 0.99; 'nights,': 0.99; 'plaza': 0.99; 'rates,': 0.99; 'rates.': 0.99; 'rooms': 0.99; 'season': 0.99; 'stations': 0.99; 'subject:Hotel': 0.99 Here is the full message (Received: headers stripped), with apologies to Ziggy and David: """ Date: Fri, 06 Sep 2002 17:17:13 -0400 From: Adam Turoff Subject: Hotel information To: guido@python.org, davida@activestate.com Message-id: <20020906211713.GK7451@panix.com> MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-disposition: inline User-Agent: Mutt/1.4i I've been looking into hotels. I poked around expedia for availability from March 26 to 29 (4 nights, wednesday thorugh saturday). I've also started contacting hotels for group rates; some of the group rates are no better than the regular rates, and they require signing a contract with a minimum number of rooms sold (with someone financially responsible for unbooked rooms). Most hotels are less than responsive... Radission - Barcelo Hotel (DuPont Circle) $125/night, $99/weekend State Plaza hotel (Foggy Bottom; near GWU) $119/night, $99/weekend Hilton Silver Spring (Near Metro, in suburban MD) $99/hight, $74/weekend Windsor Park Hotel Conn Ave, between DuPont Circle/Woodley Park Metro stations $95/night; needs a car Econo Lodge Alexandria (Near Metro, in suburban VA) $95/night This is a hand picked list; I ignored anything over $125/night, even though there are some really well situated hotels nearby at higher rates. Also, I'm not sure how much these prices reflect an expedia-only discount. I can't vouch for any of these hotels, either. I also found out that the down season for DC Hotels are mid-june through mid-september, and mid-november through mid-january. Z. """ This one has no MIME structure nor HTML! It even has a Content-disposition which is counted as a non-spam clue. It got f-p'ed because of the many hospitality-related and money-related terms. I'm surprised $125/night and similar aren't clues too. (And again, several spam clues are duplicated with different variations: 'hotel', 'hotels', 'hotels.', 'subject:Hotel', 'rates,', 'rates.'. > WRT hating HTML, possibilities include: > > 1. It really had to do with something other than MIME/HTML. > > 2. These are pure HTML (not multipart/alternative with a text/plain part), > so that the tags aren't getting stripped. The pickled classifier > despises all hints of HTML due to its c.l.py heritage. > > 3. These are multipart/alternative with a text/plain part, but the > latter doesn't contain the same text as the text/html part (for > example, as Anthony reported, perhaps the text/plain part just > says something like "This is an HMTL message."). > > If it's #2, it would be easy to add an optional bool argument to tokenize() > meaning "even if it is pure HTML, strip the tags anyway". In fact, I'd like > to do that and default it to True. The extreme hatred of HTML on tech lists > strikes me as, umm, extreme . I also looked in more detail at some f-p's in my geeks traffic. The first one's a doozie (that's the term, right? :-). It has lots of HTML clues that are apparently ignored. It was a multipart/mixed with two parts: a brief text/plain part containing one or two sentences, a mondo weird URL: http://x60.deja.com/[ST_rn=ps]/getdoc.xp?AN=687715863&CONTEXT=973121507.1408827441&hitnum=23 and some employer-generated spammish boilerplate; the second part was the HTML taken directly from the above URL. Clues: 43 1.00 S '"main"': 0.01; '(later': 0.01; '(lots': 0.01; '--paul': 0.01; '1995-2000': 0.01; 'adopt': 0.01; 'apps': 0.01; 'commands': 0.01; 'deja.com': 0.01; 'dejanews,': 0.01; 'discipline': 0.01; 'duct': 0.01; 'email addr:digicool': 0.01; 'email name:paul': 0.01; 'everitt': 0.01; 'exist,': 0.01; 'forwards': 0.01; 'framework': 0.01; 'from:email addr:digicool': 0.01; 'from:email name:1:22': 0.01; 'http>1:24': 0.01; 'http>1:57': 0.01; 'http>1:an': 0.01; 'http>1:author': 0.01; 'http>1:fmt': 0.01; 'http>1:getdoc': 0.01; 'http>1:pr': 0.01; 'http>1:products': 0.01; 'http>1:query': 0.01; 'http>1:search': 0.01; 'http>1:viewthread': 0.01; 'http>1:xp': 0.01; 'http>1:zope': 0.01; 'inventing': 0.01; 'jsp': 0.01; 'jsp.': 0.01; 'logic': 0.01; 'maps': 0.01; 'neo': 0.01; 'newsgroup,': 0.01; 'object': 0.01; 'popup': 0.01; 'probable': 0.01; 'query': 0.01; 'query,': 0.01; 'resizes': 0.01; 'servlet': 0.01; 'skip:? 20': 0.01; 'stems': 0.01; 'subject:JSP': 0.01; 'sucks!': 0.01; 'templating': 0.01; 'tempted': 0.01; 'url.': 0.01; 'usenet': 0.01; 'usenet,': 0.01; 'wrote': 0.01; 'x-mailer:mozilla 4.74 [en] (windows nt 5.0; u)': 0.01; 'zope': 0.01; '#000000;': 0.99; '#cc0000;': 0.99; '#ff3300;': 0.99; '#ff6600;': 0.99; '#ffffff;': 0.99; '©': 0.99; '>': 0.99; '  ': 0.99; '"no': 0.99; '.med': 0.99; '.small': 0.99; '0pt;': 0.99; '0px;': 0.99; '10px;': 0.99; '11pt;': 0.99; '12px;': 0.99; '18pt;': 0.99; '18px;': 0.99; '1pt;': 0.99; '2px;': 0.99; '640;': 0.99; '8pt;': 0.99; '