Re. the problems with the RSS feeds: I had temporarily set the number of comments included in the per-entry RSS feeds down to 25, and then disabled them completely, to speed up the rebuilds I had to do during the upgrade...and then I forgot to set it all back after the upgrade. It will take a little while for all of the per-entry RSS feeds to rebuild (there are many, and they have a plan), but they should be okay again after that.
@Cally #66 Can you give me the URL for the Greasemonkey script? I have no idea what's going on there.
@Raphael #91:
- The SetStyleExample.js is used for the style switcher, to make the text bigger or smaller. After the upgrade to MT4, I'd like to change this to a body switcher script (viz. Invasion of the Body Switchers and Look who's switching too), and optimize the CSS files.
- feed.js is an external script from blogads.com to display adverts in the sidebar.
- urchin.js is a Google Analytics page tracker.
- On the individual entry archives, there is an in-line script that deals with the "Don't make me type all this again" functionality: storing your email address, name, and URL in a cookie, and reading them back out again when you next visit the page.
All of these scripts are progressive enhancements, so the page works just fine without them. (For example, if you have JS turned off.)
@Earl #39 - thanks for the test case - should be fixed now.
In general, there are likely to be some small differences between the way comments show up in the normal entry archives, and on the (vab) page.
The reason for this is that the (vab) page is a custom script. It bypasses Movable Type's template and text filtering system, and rips the comments directly from the database. With its bare teeth. I've tried to match the text filters that MT applies to the comments, but I'm sure to have missed a few things.
@Clifford (#31) and @Dave (#35): as Abi mentioned in her explanatory post, adding optimized indexes for the relevant tables was the first thing we did. Also, an updated robots.txt file has been in place since the weekend.
Several people have noted that the current system doesn't deal with multiple email addresses very well. The (vab) page only shows comments from a single email address. Movable Type 3 does not have a built-in system for user profiles/user registration, so barring active human assistance to link addresses together, we have no way of knowing that monkey@example.com and fez@example.com are actually the same person.
MT4 does have user registration, so the option will be there when the upgrade is complete. Whether Patrick and Teresa want to make use of it is a completely different discussion...
The overall plan for the fixes/upgrades is roughly as follows:
- Pre-upgrade: fix the immediate performance problems, and stop leaking email addresses.
- Upgrade: migrate from MT 3.3x to MT 4.2x.
- Post-upgrade: improve spam handling, optimize templates, tweaks and tuning.
Phase 1 is now mostly complete. Phase 2 will probably start next week-ish.
For the technically minded, the major performance issue was caused by the View-all-by (vab) page, commentlist-oneauthor.php. Every time the page loaded, the underlying PHP code executed a SQL query along the lines of "select * from mt_comment where comment_email='monkey@example.com'". The mt_comment table did not have any indexes to help the execution of this query, so MySQL had to do a full table scan every time. The mt_comments table for Making Light has 300,000+ rows in it, and so the query was using up a lot of resources, regularly taking several seconds to deliver its results.
Compounding the problem was the fact that the (vab) page showed all comments for the specified commenter. For many commenters, the page size was in the multi-megabyte range. Abi's page, for example, was over 3MB. Even over a fast broadband link, that much data takes a non-trivial time to deliver, and keeps an Apache process busy for the duration.
Now, most of the requests for the (vab) script come from automated processes: bots, indexers, crawlers, spiders. Clever indexing systems, such as googlebot, keep track of how often a page is updated, and they will visit frequently updated pages more regularly. And because there is a loose correlation between how often someone posts comments here, and how many comments they have posted, the biggest pages (which consume the most resources) were being requested the most often.
The number of page requests we're talking about is substantial: the (vab) page received 4,152,243 hits in May. That's 1.5 incoming requests per second on average, for a page that took several seconds to process. The server just could not keep up. As we say in the trade, "well there's your problem."
Just for reference, Peter-Paul Koch has a set of interesting articles explaining Dutch politics for foreigners. It's an excellent overview of the system.
At least it's a grand jury. Just imagine what it would be like to serve on a petty jury for two weeks.
Our daughter is 4 years old, and attending the local Dutch primary school, where she is acquiring a traditional Noord-Hollands accent, including the aforementioned hard "g". When she says "goed" (good) it sounds like she is trying to get hork up a hairball. Breaks my heart.
Maybe it's best if you sleep in the guest room. That way you won't hear me sobbing myself to sleep.
Connie (#3): something like "DROGUE-uh VOO-tuh". Depending on what part of the country you're from, the "g" is soft and mellifluous like in the Scottish "loch", or pronounced in a guttural fashion, like someone being strangled while suffering from a catarrh-ravaged chest cold.
To elaborate slightly: my first impulse was not to vote on the basis that abstaining is a neutral position. "I don't know enough about the situation, so I'm not going to get involved."
But that argument is bogus: abstaining is not neutral. I have very strong views on a lot of matters. A vote gives me the opportunity to express those views, and to make a difference. If I don't acquaint myself with the political positions of parties standing for election, then the absence of my vote for a party may strengthen the position of one I disagree with.
Recent US elections, where hugely important matters have been determined by the slimmest of margins have really hammered that point home to me. (Note: I always voted in UK elections before we moved to NL, but the margins in recent years have generally been pretty decisive, with the result that they didn't focus my mind quite as strongly.)
Unrelated: I saw this, and thought of Halting State.
When I first saw the plastic surgery book I thought it was a a parody, because it looked so much like Microsoft's recent effort, "Mommy, why is there a server in the house?" Although the latter is clearly marketing material, at least it has the benefit of being tongue in cheek.
On the subject of interesting Christmas music, how about some Death Metal to spread a little fëstive chëer?
No-one seems to have pointed out the infamous "It's a streetlight" thread on Fark yet. Some people never let a simple explanation get in the way of a perfectly good whacked-out theory.
I've just been thinking about the similarities between programming and writing in this context. Writers have editors, and gather rejection slips; programmers have testers, and gather bug reports. Neither should be taken personally, because they're not intended as such. In fact, there's a technical term for coders who take bug reports as personal affronts: project managers. (Miaow.)
The 13 reasons for rejection also map roughly to the reasons a tester might have for not accepting a piece of software:
- 1-7: code doesn't compile
- 8-10: code doesn't conform to requirements
- 11-13: code doesn't run fast enough
The software teams that finish their projects and actually ship product are the ones that buckle down and keep working until they get to 14: acceptance. Same with writers.
| Year | Number of comments posted |
|---|---|
| 2009 | 9 |
| 2008 | 5 |
| 2007 | 2 |
| 2004 | 1 |
Total: 17 comments. View all these comments on a single page.
The most recent 20 comments posted to Making Light by Abi's Husband:
Show all comments by Abi's Husband.