Return-Path: tim.one@comcast.net Delivery-Date: Mon Sep 9 17:46:55 2002 From: tim.one@comcast.net (Tim Peters) Date: Mon, 09 Sep 2002 12:46:55 -0400 Subject: [Spambayes] deleting "duplicate" spam before training? good idea orbad? In-Reply-To: <15740.52432.861148.597750@12-248-11-90.client.attbi.com> Message-ID: [Skip Montanaro] > Because I get mail through several different email addresses, I > frequently get duplicates (or triplicates or more-plicates) of > various spam messages. In saving spam for later analysis I haven't > always been careful to avoid saving such duplicates. > > I wrote a script some time ago to try an minimize the duplicates I see > by calculating a loose checksum, but I still have some duplicates. > Should I delete the duplicates before training or not? People just can't stop thinking . The classifier should work best when trained on a wholly random spattering of real life. If real life contains duplicates, then that's what the classifier should see. > Would people be interested in the script? I'd be happy to extricate > it from my local modules and check it into CVS. Sure! I think it's relevant, but maybe for another purpose. Paul Svensson is thinking harder about real people than the rest of us, and he may be able to get use out of approaches that identify closely related spam. For example, some amount of spam is going to end up in the ham training data in real life use, and any sort of similarity score to a piece of known spam may be an aid in finding and purging it.