Go to previous post:
Staten Island ferry crash.

Go to Electrolite's front page.

Go to next post:
User unsatisfactory.

Our Admirable Sponsors

October 15, 2003

Solution Unsatisfactory. Yes, we’re aware of Jay Allen’s MT-Blacklist, the Movable Type plugin that consolidates and simplifies a whole bunch of different approaches to stopping weblog comment spam. We even recommend it. Try it out. It’ll probably work for you.

Unfortunately, it doesn’t work for us. Quite the contrary, it slows routine rebuilds to a crawl, and the “despam” link it adds to our comment-notification email leads to a Movable Type error message. We’ve been in touch with the estimable Jay Allen about this, and he’s made a valiant effort to figure out the problem, but no dice.

The last straw, though, was when Teresa attempted to post a comment to a thread on her own weblog and got the following error message:

Your comment submission failed for the following reasons:

This comment could not be posted due to questionable content.

Please correct the error in the form below, then press POST to post your comment.

After peeling one (1) outraged spouse off the ceiling, I deleted MT-Blacklist from our Movable Type installation, and the comment posted successfully. You can read it here; it’s the one that begins “When are we going to get serious about spammers? Do you ever think about how much human effort gets pissed away dealing with them?”

Yes, God is an iron, why do you ask?

I had no idea MT-Blacklist was filtering based on mere content; I thought the idea was to look for—and filter based on—the spammy URLs that blogspammers are trying to get onto our sites. Indeed, as you can see, Teresa’s comment contains no URLs whatsoever.

I think we’re giving up on MT-Blacklist for now. Whether we give up on blogging altogether is a still-unanswered question.

UPDATE: If you’ve come to this post from an external link such as this one, be sure to read the followup post as well. Things are nowhere near so dire. [10:51 PM]

Welcome to Electrolite's comments section.
Hard-Hitting Moderator: Teresa Nielsen Hayden.

Comments on Solution Unsatisfactory.:

Ginger ::: (view all by) ::: October 15, 2003, 11:01 PM:

Count me among the many people who will be sad if you guys quit.

One of the solutions we're currently considering is closing comments on all entries older than [time to be decided]. There's a plug-in or hack that does it if you're using PHP. Another alternative we're considering is a cron job that changes the flag in the database and rebuilds the blog overnight.

This won't solve the problem of spam on current entries, but it will solve the problem of spam on hundreds of older ones.

Bruce Baugh ::: (view all by) ::: October 15, 2003, 11:35 PM:

I would hate to see your folks take your weblogs down. Not that you owe it to any of us, but that you give us information and entertainment and stimulation and sometimes just the right kind of outrage.

I suspect that I'm not alone in being willing to do some hoop-jumping for the opportunity to continue commenting from time to time, whether that's a registration system plugged into MT comments, a separate forum (I've seen some weblogs attached to PHPbb forums), or whatever.

Avram ::: (view all by) ::: October 16, 2003, 12:53 AM:

I just downloaded the source and gave it the once-over. I'm not sure, but I think it'll filter based on arbitrary strings. I'd have a better idea if I sat through an install, which isn't an option for me, not having Moveable Type handy.

Suw ::: (view all by) ::: October 16, 2003, 03:22 AM:

I know how tedious web communities can get when you have even but one person intent on ignoring the rules of decent human behaviour, but I know I'd miss Electrolite if you were to close it.

Maybe hive off the comments to a separate forum, or move the blog to a different provider. I know both of those possibilities would cause you hassle, but I'm sure I'm not alone in saying that I appreciate your blog and the effort that goes into maintaining it and keeping it spam free. If there was anything I could do to help I'd be happy to.

bry ::: (view all by) ::: October 16, 2003, 06:10 AM:

solving this kind of problem is rather old, solving it in the new context is rather new. I suppose that the various people doing blogging tools have not had to much experience in the area before, therefore most early solutions are gonna be lousy (just a general statement, I know nothing about Movable Type). filtering on strings is unworkable really, unless you have a bayesian system, that would probably (read almost certainly, for reasons given at the end) slow the system down too. Any system that gave the text shown above a false positive is bad though, I would think that mentioning spam in a comment is not something most likely for a spammer to do. The purpose of comment spam is to put in urls, if there is no url put in it is not spam (that should be the first test - I am actually working on this problem for a product my company's releasing later this year), furthermore a really good system (if it could be kept fast enough) would have the concept of trusted posters, name, i.p, and email combos that are trusted and thus never kept from posting, such trust should be possible to be set by administrators or, and I find this preferable, learned by the system as it goes on.

What would be the likely behavior of a trusted poster, 1. trusted posters don't always post links

2. trusted posters don't always post links to the same domain.

3. trusted posters carry on communication with other trusted posters. By which one could say that a post by Teresa (who is trusted) mentions the post by Xopher (who is not yet trusted) and quotes from it. This puts xopher on a shortlist by the system to get trusted.
Further communication between trusted posters in a thread and across multiple threads with posters that are not trusted increases the chance that those posters should be trusted.

Frankly the problem here is the scalability of any solution, the blogging community being what it is, and the costs of the products sold to that community being what they are, it does not seem likely to me that any blogging tool would have the muscle needed to actually handle that kind of problem (a blogging service however, in conjunction with usage of tools, should be able to provide said muscle).

Anyway that was just my two cents, prompted by a problem that I am somewhat considering at work right now (as a side matter really.)

Kathryn Cramer ::: (view all by) ::: October 16, 2003, 07:55 AM:

Whether we give up on blogging altogether is a still-unanswered question.

Come down from the ceiling in the name of the Lord. (Or am I misquoting the line from a long ago Little Mag manuscript?)

Perhaps we could use a MT plugin that has a special type of delete in which a screen comes up allowing one to contact law enforcement and/or one's elected representatives with the full info on the spammer ever time one has to delete such a thing.

I suspect that the posting of porn links to the web site of another may be illegal under some existing law. And inasmuch as it isn't it should be and could be. This is, free speechwise, an easier problem than email.

Teresa Nielsen Hayden ::: (view all by) ::: October 16, 2003, 09:07 AM:

We were tired and dispirited last night, and I'll admit that my reaction to being told that my comment had "questionable content" and therefore couldn't be posted to my weblog was not entirely reasonable. It was just software doing something I didn't want. It didn't actually constitute an act of le8se-majeste9. I knew that. I just had to work on believing it.

My response was not unreasonable; I told Patrick what was going on, saved the comment to a text file, and eventually got it posted. That was good. But it took a while for the incandescence to subside.

"Come down from the ceiling in the name of the Lord" may or may not be the line from that ms. -- it's been a while -- but it's the right line to use on me. If Madeleine had had it in her arsenal back when she was my assistant, she would undoubtedly have used it.

Jay Allen ::: (view all by) ::: October 16, 2003, 11:14 AM:

It happened because of a bug in the default blacklist. See here . My deepest apologies to you, Teresa. MT-Blacklist IS beta software. There will be (and ARE!) bugs. I and my alpha test team are doing the best we can to balnace making great, bug-free software and the immediate and desperate needs of the community.

That said, sometimes I get really mad at my computer too... :-)

Bill Higgins-- Beam Jockey ::: (view all by) ::: October 16, 2003, 11:15 AM:

TNH writes in response to Kathryn Cramer:

"'Come down from the ceiling in the name of the Lord' may or may not be the line from that ms. -- it's been a while -- but it's the right line to use on me. If Madeleine had had it in her arsenal back when she was my assistant, she would undoubtedly have used it."

There's a story here, which I have not been told.

Alan Bostick ::: (view all by) ::: October 16, 2003, 11:30 AM:

There so very often is, in my experience.

Patrick Nielsen Hayden ::: (view all by) ::: October 16, 2003, 11:51 AM:

Jay, thanks again. Indeed, I picked up the regexp correction from your weblog here, even before I noticed you'd been kind enough to come over and bring it to our attention.

I'm also delighted to have discovered this easy fix for the slow-rebuild problem; in essence, by commenting out three lines of code, we got our fast rebuilds back, and the only functionality we've lost is MT-Blacklist checking all our old comments against the blacklist. Since we already deleted most of the spam comments in the database before installing MT-Blacklist, this check isn't the most important feature to us.

Teresa Nielsen Hayden ::: (view all by) ::: October 16, 2003, 12:23 PM:

Jay Allen, I know it's beta software, and I apologize for forgetting that. You're being heroic.

Bill, that notable novelist Madeleine Robins, former Assistant to the Publisher at Tor, was before that the Assistant to the Managing Editor, who was me. She was a marvelous assistant. Among her many virtues was the ability to talk me out of my occasional bouts of High Dudgeon. I don't have those often, and they do pass; but they don't always pass fast enough. This is less than optimal, because when I'm in that state I tend to want to Do Something about whatever's set me off.

Rebooting my senses of proportion and humor are usually enough to do it. "Come down from the ceiling in the name of the Lord" would do both, and Madeleine would undoubtedly deliver the line with great panache.

As for that long-ago manuscript sent to The Little Magazine, I can only plead that while that one line sticks in memory, I read the piece so long ago that I can't give a fair description of the rest of it. This happens. I have only a dim recollection of a batch of odd pungent poetry we got from some other guy around that time, but one of his lines stuck with me: "When someone tells you they're a lousy lover, they're always right."