Also, it's a good idea to zip some of those backups and toss them onto someone else's server for safekeeping -- even if the new server is rock-steady and RAIDed.

Bravo, all.

The photograph that terrorized London: The word "March" in the first sentence is highlighted.
Greyhawk's flags at half-staff: The apostrophe in the title has turned into an underscore.

Fulll-site rebuilds are still failing four times out of five.

I have cached copies of comment RSS feeds for the following post numbers; if you still have use for them, let me know and I'll upload them.

8696 8724 8735 8751 8756 8759 8772 8777 8786 8804 8806 8810 8811 8833 8839 8843 8844 8845 8851 8856 8858 8859 8860 8862 8867 8868 8883 8884 8890 8893 8901 8902 8912 8913 8914 8929 8931 8932 8934 8936 8938 8948 8949 8951 8952 8953 8966 8991 9015 9017 9020 9023 9025 9027 9034 9050 9052 9062 9072 9076 9077 9081 9082 9083 9084 9086 9087 9090 9092 9093 9095 9096 9105 9107 9116 9117 9132 9137 9143 9144 9148 9150 9156 9165 9170 9176 9180 9186 9187 9200 9201 9203 9211 9215 9220 9221 9226 9233 9234 9238 9239 9245 9256 9262 9263 9268 9273 9274 9280 9283 9286 9289 9290 9291 9295 9302 9304 9305 9322 9323 9327 9328 9329 9333 9334 9335 9337 9339 9342 9360 9378 9381 9387 9390 9401 9402 9409 9418 9421 9426 9434 9435 9436 9437 9446 9448 9454 9460 9461 9470 9474 9478 9479 9484 9493 9494 9501 9511 9519 9524 9527 9528 9534 9536 9537 9542 9544 9546 9559 9593 9595 9615 9620 9648 9653 9654 9665 9673 9676 9679 9680 9682 9689 9690 9694 9743 9746 9747 9755 9756 9759 9791 9801 9803 9804 9805 9806 9808 9819 9820 9821 9822 9823 9826 9832 9835 9837 9843 9852 9857 9863 9872 9881 9883 9884 9885 9886 9890 9892 9893 9897 9900 9901 9904 9905 9906 9910 9914 9918 9931 9932 9933 9935 9938 9941 9942 9951 9953 9957 9966 9967 9971 9980 9982 9983 9988 9994 9995 9997 9999 10002 10003 10010 10012 10025 10028 10044 10046 10066 10069 10077 10097 10101 10104 10108 10113 10115 10123 10125 10129 10133 10134 10142 10143 10146 10151 10154 10157 10173 10174 10175 10176 10177 10178 10184 10186

Mary Dell, #1: You're absolutely right. Indeed, no sooner did I read your comment than I went into the other room and performed a MySQL database export. Now the desktop machine is, via sftp, making a complete copy of the home directory.

(We are all clear that although the Nielsen Haydens live in Brooklyn, Making Light lives on a server at a NOC in Jacksonville, Florida? Right then.)

Someone was mentioning that it's reasonably simple to set up a MySQL database so that changes to the DB (such as, in our case, adding a new post, comment, etc) are automatically made to a copy of the DB hosted elsewhere. This seems like an awfully good idea and I'd like to get something like that set up. Along with, of course, automated daily backups of the site itself.

Sorry, I'm not certain how to read that. Do you want the data? I find "probably" and "nearly all" rather alarming, and I'd like to help.

Assuming no bizarre glitches, I have complete copies of these threads up to some time at most 24 hours (my update frequency) before the failure. I have a comment dated Fri, 02 May 2008 15:54:27 -0500.

http://209.85.135.104/search?q=cache:1-Y6js2ED9oJ:nielsenhayden.com/narcolepsy-observer.html

http://209.85.135.104/search?q=cache:nielsenhayden.com/bigz.html

(I'm just

Also, if your server is 3rd-party hosted, they should be backing it up as well, and should have a doc stating exactly what they do and what kind of turnaround you can expect, blah blah. Not all ISP's do that, though, obviously.

(mentioned in one of the trillion emails I've flooded you with: if you want me to vet out your options & choices for data protection & service continuity, I'd be happy to)

*Florida's not necessarily a bad location for servers, but depending on where the NOC is and how their power is handled, there may be more power outages than would occur in a less storm-prone location.

Want to hear the really funny one? While Patrick and Abi and I were working so intensively on saving everything else, saving our own comments completely slipped our minds. Jim Macdonald foresaw that and saved them all for us, and Avram Grumer's as well.

I miss the old net.practice of mirroring.

"We ain't got no backups. We don't need no backups. I don't have to show you any stinking backups!"

That reminds me of the big 1991 fire in the Oakland Hills, and reading about how a writer (Maxine Hong Kingston?) lost a novel because she had no backup. I back up my wife's stuff every evening. Probably a wise decision, what with her HP Pavilion's reliability having become a tad erratic.

That being said, my felicitations to all of the people who worked to recover ML's threads and comments.

http://worldcommunity.com/opensource/utilities/mysql_backup.html

Not terribly secure, but it's pretty much all public data anyhoo.

I can still use these, though, as well as Chris's data -- if there are comments not included in our many HTML copies of threads, we can take whatever leftovers are there, and make an educated guess on the identity of the poster. In most cases, of course, that'll be sufficient.

I'll be doing comments over the course of the day. They're not hard, just tedious. And copious.

In re supposed glitches -- I thought the Google highlighting gave the restored posts a general air of world-weary character, as though they'd been around the block a couple of times and were happy to be home.

Put me down as willing to do some grunt-work. I am reasonably computer proficient, and can execute specific instructions well.

Contact info is the lj name in the link, at juno.com

But the site is back up, which is more than enough.

robots.txt: stick in root directory of webserver, with the contents:

User-agent: *
Disallow: /

More info at: http://en.wikipedia.org/wiki/Robots.txt

I'm in awe, jaw-dropped awe, at what you have accomplished, and in such a short time, in terms of this site catastrophe. What brilliant commentators you all have (so NOT including moi in that category!).

I can only imagine. Upon arriving 'home' I went directly to the NYPL "Books and Materials for Checkout" OPAC -- and it was GONE. No explanation either, unlike you guys.

Love, C.

The realization of that fact is part of what sent me on my quest late Saturday--wanting to work ahead of the spiders, or the little blue workmen of Google (obsftv: 'Matter of Minutes').

And that was fun.

I hadn't been in a state of Mihaly-Vingean Flowcus for a while, and having a brief Algernon-peak-experience about here--all the people, all the poems--was nifty.

It's my impression that if you disallow all, it will drop you from the caches instead of overwriting with older data.

It seems to me that there should be a best practices set of guidelines for what to do to your site while trying to recover things from various internet caches. -- Old stuff: bad, overwrites caches. Robots.txt: bad, clears caches. ??? good, tells google that there are problems and come back later.

I'm thinking that a 4xx series response of internal server error - content missing may be appropriate, It's a temporary error, indicating that a retry is appropriate at a later date.

01010100101010110101110000110110101011001011101000111

(I replied to Jim but it seems to have vanished, so a rewording:)

You did see that there; that database isn't reloaded yet. I caught a cache of the Particles and Sidelights through 4/30, which I believe was the last update, and sent it to P & T.

Hey look, it's people on the Internet spontaneously ordering themselves into productive labor! Eat your heart out, Clay Shirkey!

Is the HTML "anchor" used on individual posts, i.e. the part after the '#' in the URL, a simple unique identifier incrementing per-post across all threads? It looks to me like it is, but I can't definitively tell.

This would be very important to know, if so.

The reasons are several:

1. it would provide a handy key for identifying duplicate copies of the same post from different collaborators when reassembling the database.
2. it would let us put back each post in its correct relative place;
3. it would let us tell whether posts are missing and where in sequence;
4. it would let us know where placeholders might be needed for missing posts, and what we should be searching for.

If the anchor does act as a unique global key, this (not hexapodia!) might be an important insight to eliminate most of the work of retting, hackling, and spinning, as Teresa said. Eli's cotton gin, here we come!

http://nielsenhayden.com/makinglight/archives/cached-007399.html

and the current page:

http://nielsenhayden.com/makinglight/archives/007399.html

and confirm to me that there are posts missing in the middle of the page currently online?

However, comments are always attached to their time of posting, and that preserves order (to within a minute -- not a second, as I mistakenly posted elsewhere.)

So what I'm doing is looking at email-plus-posting-time as a "key" and refusing to load comments that would collide on that basis. But on 007399 (the first post I've tried this on) this is giving me confusing results. It didn't help that I was originally looking at 00740something for some reason. (The damage I could do if I ever had enough sleep! I blame the kids.)

Note that the page went dead from April 10, 2007 (comment 223) to April 21, 2008 (comment 224).

I see no evidence that the cached page is missing any comments.

Nope, it looks to me* like you have a good solid run in the currently-online page, that ends at #223. The missing parts come after that.

It also appears that a random sampling of posts from the currently-online page and the cached page shows the same comment numbering for both - eg, Jim's phrase "A naïve T-cell or B-cell" appears in post #218 in both versions; Teresa's "I capitalize Creationist the same way I would the name of any other religious cult" is #109 in both, etc.

*(based on skim-n-sample, not a line-by-line check)

It might not be a bad idea to put a note there, saying that the crisis is over, and directing people back here.

The first few having the "December, 0, 12: AM" dates suggests that perhaps some script misparsed the date strings on some of the cached posts, and that threw off the sequence.

Computers always do exactly what you tell them to do; the trick is figuring out whatintheheck it was you told them.

You know we are all fallen at your feet in awe at what you've achieved, don't you? Just sayin'.

Sorry it took so long; there was a sudden and traumatic discovery of a forgotten sack of mangoes on the balcony -- I knew I was smelling vinegar over there yesterday -- with a concomitantly large population of Drosophilia melanogaster. What really made it exciting is that they had been stacked on top of a box of photos. (A plastic box, fortunately -- but still...)

No damage, fortunately. Except to the D. melanogaster, I'm sad to say.

Abi: aw, shucks, 'tweren't nothin. And would have all gone much faster if I'd used Arachne's code in the first place -- ah, well.

Two threads down. Many to go.

A friend of mine and I, who were in business together doing technical stuff, used to joke that programming and sysadmin work is the only field where we could definitely imagine what it might be like to be smarter.

Just the variation between better days and offer days is enough to make that clear.

There's a happily unused column called "created_by", into which I've written the value "fiat" for the restored posts. This is a handy flag for seeing which ones should be deemed suspicious. But I forgot to specify "fiat" when removing comments for 106. Sorry.

It always seemed like the hardest part of anything involving MySQL was making sure it and any associated programs were using the same notion of time; just negotiating the differences between GMT, CST and CDT appeared sometimes to be almost insuperable. So forgetting noon vs midnight looks almost quotidian by comparison. It's not smarts, it's just keeping track. Totally else different thing.

(I once deliberately raised a colony of D. melanogaster in my bedroom. The problem arose when they escaped to the kitchen. My mother was not best pleased.)

I've got some of the posters who mostly stayed on the salwar kameez thread (which may have been saved anyway), and a variety of other stuff.

These are the posters I have on the salwar kameez thread, and their dates:
lois, 3-29
anne bremser 3-22
maggie 4-7
star 2-28
litlnemo 2-15 already there
dee 4-4

These are other random posters I have saved, and their dates:
nicole j leboeuf-little 4-26
nicole twn 4-15
nicole the wondernerd 4-15
lois fundis 4-25
carrieS 4-30
susan dancing 5-2
albatross 5-2

If you want to pick up where I left off, checking old threads for new posts, start here:
http://nielsenhayden.com/makinglight/archives/009764.html
Feed that link into the search engine at:
http://www.live.com
Click on "cached page" and look at the date of the last comment of the thread. If it is after February 29, 2008, save the page as a "web page, complete," compress it, and mail it to Patrick.
Click on the previous post link of the page you just examined, and repeat for the new post!

I need to take a break, and attend to some physical life things. If you decide to do this, please post that, and when you stop, the link you leave off at, so that I can pick up when I get back.

I has lolcatz thred. Ask if want. kthnxbai!

However, it's now been replaced by a Dina Shunra's comment, which actually continues the thread's discussion!

This is fun! I've never done data work in a fishbowl before -- it's totally cool being able to ask, hey fluorosphere, does this look right? It's like ... like ... the Singularity or something.

I'm going to be scarce for a couple of days. Tomorrow we're driving up to San Juan to stay overnight (my daughter is representing Puerto Rico at the national Mathcounts competition in Denver this weekend -- and going alone with the team! Yikes! At 13!) So she'll fly out bright and early Wednesday, after which we'll take our son to El Yunque (the only tropical rainforest in the national parks system.) And then it's back home again and back to work on Thursday.

Various posters (mostly A-C) plus some random others whose posts I saved last night and the dates:

adam ek through 4-28-08
adamsj through 5-1-08
ajay through 4-30-08
alan bostick through 4-1-08
alan braggins through 4-16-08
alan yee through 3-29-08
albatross through 4-28-08
alex-cohen through 4-25-08
anna feruglio through 4-14-08
bill higgins through 5-1-2008
bob rossney through 5-1-2008
bruce cohen (speakertomanagers) through 5-2-2008
cajunfj40 through 4-30-2008
carol kimball through 4-30-2008
carrie s through 4-30-2008
chris quinones through 4-29-2008
chris turkel through 4-18-08
clifton royston through 5-2-2008
connie h through 4-28-2008
cory doctorow through 4-24-2008
dena shunra through 4-23-2008
heresiarch through 4-30-2008
jo walton through 4-16-08
linkmeister through 4-23-08
marilee through 4-30-2008
mary dell through 5-2-2008
susan (de g) through 5-2-08

You're doing quite awesome work!

Don't die!

Hearts and flowers!

Now, back to work for me. I'll be more participatory next week.

Excelsior First Officer: [over intercom] Yellow Alert! Yellow Alert!
Captain Styles: Bridge, this is the captain, how can you have a yellow alert in spacedock?
Excelsior First Officer: Sir, someone is stealing the Enterprise!

Sending the zip to PNH this morning.

More seriously, I'm here to mention the prevalence of line break characters in some of the threads, particularly "Teresa in the Observer". I haven't checked others, but I suspect that some of the restored comments will have them in.

Back to cataloging.

The new phone books are here! The new phone books are here!
- The Jerk

Is posting this sort of thing here helpful?

Hope this isn't a repeat catch -- just flying through and wanted to read the original version.

Nope, that was a mis-post originally. (He reposted it in the right thread a few minutes later, and that post is in place.)

Well spotted, though.

Also, Charlie Stross noted at OT107 that "strikeout" fell out of the allowed HTML tags in comments. While on that topic, could we have a a complete list of the allowed tags? It wouldn't take much space if done like this:

<strong>Strong<strong>, <b>Bold<b>, <em>Emphasized<em>, <i>Italic<i>, <s>Strikeout<s> <blockquote>Blockquote<blockquote> ...

Hmm, you're apparently not permitting "big", "small", or "q)uote" styles. The "quote" style might be worthwhile to allow along with "blockquote". Small-caps might be cool too, if only for Pratchett riffs.

Looks like "none of the above". And on further reading, it seems <del> is the currently-preferred way to get the strikeout effect.

Hmm... < foo > Foo < / foo > vs. <foo>Foo</foo>

No, that works... bah, iduno.

Aha. Well then I guess it's just where it ought to be, eh?

(I got there following a link from an ancient BB post)

They still haven't reappeared on the site. Is this a sign that I did something wrong, that I need to have more patience, or what?