Back to previous post: The shoes of Nuestra Señora de Guadalupe

Go to Making Light's front page.

Forward to next post: In Brooklyn, about a mile south of us

Subscribe (via RSS) to this post's comment thread. (What does this mean? Here's a quick introduction.)

June 13, 2009

Service Advisory
Posted by Patrick at 01:59 PM * 84 comments

We’ll be implementing the long-awaited upgrade to Movable Type 4 tomorrow morning. As a result, for about three hours, commenting will be disabled. The front page and individual archive pages should all remain readable.

Assuming all goes well (knock wood, touch a belaying pin, throw a goat over one’s left shoulder, bury a statue of Giordano Bruno upside down) (the best superstitions are the ones you make up yourself), the no-commenting period should last about three hours. It’ll begin at 9 AM European summer time, which is 8 AM in the British Isles, 3 AM on the east coast of North America, and midnight on the west coast. The rest of you can work out the time differences for yourselves. Don’t forget the goat.

Update from Martin at 12:30 CET: This is not going as well as I had hoped. Commenting is still down; investigation ongoing.

Update from Patrick at 8:30 EDT: While Martin investigates, he’s reverted us to MT 3.33, and commenting has been re-enabled. We’ll try again when he’s reasonably sure it will work. Carry on…

Comments on Service Advisory:
#1 ::: janetl ::: (view all by) ::: June 13, 2009, 02:16 PM:

I assume that it's a coincidence that you're doing this on Sunday the 14th, rather than Saturday the 13th?

#2 ::: Jon Meltzer ::: (view all by) ::: June 13, 2009, 02:17 PM:

Are you sure a goat will be enough? A bull might be safer.

#3 ::: Bruce Adelsohn ::: (view all by) ::: June 13, 2009, 02:28 PM:

Let's leave the goat for LJ, shall we?

A llama, on the other hand... :-)

#4 ::: Xopher ::: (view all by) ::: June 13, 2009, 02:28 PM:

janetl, "Flag Day" used to be slang in the software industry for a big change done all at once, after which users have to get used to the new system. Usually involves some downtime.

Whether it's a coincidence or not, it's fortuitous, isn't it? Writing this before any answer has been given, but it is kind of cool.

Patrick, I've often declared that rubbing a bald man's head is good luck. I may have gotten that from somewhere, but I promote it because I like it.

#5 ::: James D. Macdonald ::: (view all by) ::: June 13, 2009, 02:29 PM:

There's enough bull to go around. We just have to be careful that no one gets our goat.

#6 ::: Fragano Ledgister ::: (view all by) ::: June 13, 2009, 02:40 PM:

Might I enquire as to whether the goat should be a Nubian or an Arabian? These things matter.

#7 ::: abi ::: (view all by) ::: June 13, 2009, 02:45 PM:

Xopher @4:
We always called those "Big Bang" implementations. But we're a tad late to do it on that day, no matter what else.

I could tell you why it's to be tomorrow, but it's much more fun to see everyone speculate, to be honest.

#8 ::: Jon Meltzer ::: (view all by) ::: June 13, 2009, 02:47 PM:

#6: and, what is the goat's air-speed velocity?

#9 ::: KristianB ::: (view all by) ::: June 13, 2009, 02:47 PM:

Sadly I only have the one statue of Giordano Bruno, so I have to retrieve it every time once the event has passed.

#10 ::: Debra Doyle ::: (view all by) ::: June 13, 2009, 03:50 PM:

Fragano@6: A Nubian for hardware upgrades, and an Arabian for software upgrades.

For changes to the user interface, a sheep.

#11 ::: Serge ::: (view all by) ::: June 13, 2009, 03:55 PM:

If this were an episode of Eureka...

#12 ::: Linkmeister ::: (view all by) ::: June 13, 2009, 04:00 PM:

Xopher, "I've often declared that rubbing a bald man's head is good luck. I may have gotten that from somewhere. . ."

Um. That was GWB's gesture.

Not to say you appropriated it from him.

#13 ::: Fragano Ledgister ::: (view all by) ::: June 13, 2009, 04:11 PM:

Debra Doyle #10: A black-belly sheep for customer support?

#14 ::: joann ::: (view all by) ::: June 13, 2009, 04:16 PM:

Or perhaps a rubber chicken? Spouse keeps one in his office, attached to a toilet plunger handle, and spins it round over his head when things get really dicey.

#15 ::: janetl ::: (view all by) ::: June 13, 2009, 04:35 PM:

Xopher @ 4: I have heard quite a variety of different terms for a scheduled ship date, but I'd never heard "flag day" before.
As to good luck talismans, we have a piece of 2x4 sitting on the bookcase in the area where we have our daily standup. Whenever anyone says anything that might tempt the gods to punish us for hubris, we knock on it. It recently acquired a sparkly, pink unicorn sticker. Depending on your point of view, this may enhance it's effectiveness, or not.

#16 ::: joann ::: (view all by) ::: June 13, 2009, 05:14 PM:

janetl #15:

It's not a ship date, it's a change date. Usually involves heavy-duty maintenance and/or systems tweaking. Not what you're sending to customers, but rather what you're doing to yourself.

#17 ::: James D. Macdonald ::: (view all by) ::: June 13, 2009, 05:14 PM:

"Don't worry, the change will be transparent to the users...."

"Much like a helicopter's rotor blades."

#18 ::: Serge ::: (view all by) ::: June 13, 2009, 05:15 PM:

Xopher @ 4... I've often declared that rubbing a bald man's head is good luck

It is a truth universally acknowledged that nothing but very good luck can ensue from rubbing the not-quite-bald head of a French-Canadian who sounds like Christophe Lambert.

Me fishing for head rubs?

Jamais!

#19 ::: joann ::: (view all by) ::: June 13, 2009, 05:17 PM:

Serge #18:

Jamais direz-vous jamais!

#20 ::: Serge ::: (view all by) ::: June 13, 2009, 05:22 PM:

joann @ 19... Non, rien de rien, non je ne regrette rien.

#21 ::: Serge ::: (view all by) ::: June 13, 2009, 05:24 PM:

James McDonald @ 17... Did that one originate with you? No matter what, next time someone tells me "this is going to be totally transparent", I'll have to quote you.

#22 ::: xeger ::: (view all by) ::: June 13, 2009, 05:44 PM:

I'm reminded of a recent outage notification which started with "unplanned" and included as the cause "unplanned" ;)

That said, may your goats be entertaining (perhaps fainting ones, and a few handy loud noises), and there be no need for the invocation of cowboys to resolve a rodeo situation...

#23 ::: Wyman Cooke ::: (view all by) ::: June 13, 2009, 05:46 PM:

Serge @ 11: Who's to say it won't be?

#24 ::: Serge ::: (view all by) ::: June 13, 2009, 06:02 PM:

Wyman Cooke @ 23...

"So. Invisible killer drones."
"Yeah. Keeps getting better."

#25 ::: David Harmon ::: (view all by) ::: June 13, 2009, 07:21 PM:

Linkmeister #12: Rubbing the head of bald guys was around long before Shrub! Remember Steve Austin on the Jerry Springer show? It's more likely derived from customs surrounding Buddha statue(tte)s.

#26 ::: David Harmon ::: (view all by) ::: June 13, 2009, 07:27 PM:

Also, instead of Bruno, shouldn't you be using a statue of Charles Babbage? Or even Alan Turing?
:-)

#28 ::: LMB MacAlister ::: (view all by) ::: June 13, 2009, 07:56 PM:

Damn! I read Patrick's post on the front page, and wanting to do anything I could to help, I sprang into action. Now I see I read too quickly, and need to head outside and dig up Bruno Giordano!

I hope he's still breathing.

#29 ::: Rob Rusick ::: (view all by) ::: June 13, 2009, 09:03 PM:

David Harmon @25: Rubbing the head of bald guys was around long before Shrub!

I understand there is a bronze plaque at MIT of George Eastman, with a well polished nose. It has been the tradition for students to rub his nose for luck before taking an exam, and the thousands of test-taking rational engineers have kept George's nose buffed to a shine.

#30 ::: Terry Karney ::: (view all by) ::: June 13, 2009, 09:29 PM:

abi: Y'all are doing it for my birthday. D'uh (which I share, with the flag, as mentioned and the Army).

#31 ::: Terry Karney ::: (view all by) ::: June 13, 2009, 09:33 PM:

Re statue rubbing. The nose of Rickover is bright for the saame reason. The statue of the Naval Acadamy has a region kept polished by sending plebes to caress it, and the nipples on the mermaids at the MGM (which were made by a friend of mine) have had to be repatinad several times, and repaired once.

#32 ::: P J Evans ::: (view all by) ::: June 13, 2009, 09:48 PM:

At UCSB there was a bust of Storke which had a polished nose, for the same reason. It was in the UCen (essentially the student union, which at the time included the bookstore and university branch post office).

#33 ::: Soon Lee ::: (view all by) ::: June 14, 2009, 12:15 AM:

Fingers crossed that it will/has go/gone well.

#34 ::: Lee ::: (view all by) ::: June 14, 2009, 12:44 AM:

Fragano, #6: Would that be an African or a European swallow?

#35 ::: Wyman Cooke ::: (view all by) ::: June 14, 2009, 01:08 AM:

Knock wood that it will be 3 hours and not 24-48 before ML is back on the intertubes. Kthnxbai!

#36 ::: Wyman Cooke ::: (view all by) ::: June 14, 2009, 01:15 AM:

Knock wood that it will be 3 hours and not 24-48 before ML is back on the intertubes. Kthnxbai!

#37 ::: Wyman Cooke ::: (view all by) ::: June 14, 2009, 01:16 AM:

Plz delete the duplicate. Kthnxbai!

#38 ::: Bill Higgins-- Beam Jockey ::: (view all by) ::: June 14, 2009, 02:22 AM:

Could "making up a new supersition yourself" be good luck?

#39 ::: John Chu ::: (view all by) ::: June 14, 2009, 07:54 AM:

Hey, comments are back. Congrats.

#40 ::: Serge ::: (view all by) ::: June 14, 2009, 08:29 AM:

There should have been an earth-shattering kaboom. Where was the earth-shattering kaboom?

#41 ::: Patrick Nielsen Hayden ::: (view all by) ::: June 14, 2009, 08:34 AM:

More like a damp squib. We're looking into it; meanwhile, as my update said, carry on.

The next attempt may entail somewhat less notice.

#42 ::: Andrew Plotkin ::: (view all by) ::: June 14, 2009, 09:15 AM:

Carnegie Mellon also has a plaque with a shiny nose... except I never saw a student rub it, and I *did* see maintenance staff polishing the thing on their late-night rounds. So, an artificially-maintained good-luck tradition, exempli gratia.

#43 ::: abi ::: (view all by) ::: June 14, 2009, 09:26 AM:

Andrew Plotkin @42:

An alternative explanation is that the nose is lucky, not for students, but for maintenance staff?

#44 ::: Serge ::: (view all by) ::: June 14, 2009, 09:31 AM:

No matter why the shine, today's news were a good nose. My many thanks to Martin, Abi and everybody else.

#45 ::: xeger ::: (view all by) ::: June 14, 2009, 10:31 AM:

Serge @ 44 ...
It's allergy season ... no nose would be good nose...

[but yes ... most excellent news!]

#46 ::: abi ::: (view all by) ::: June 14, 2009, 10:40 AM:

Guys,

The congratulations must be returned with thanks. Read Patrick's note again.

The upgrade failed; we fell back to the previous version. We're going to be investigating why and trying again at another time.

#47 ::: Serge ::: (view all by) ::: June 14, 2009, 10:45 AM:

Abi... Oh. I guess I'm so used to panicky it-didn't-work!'s here at the office that it didn't register as a curses-foiled-again. Well, look at the congrats as an advance payment.

#48 ::: Serge ::: (view all by) ::: June 14, 2009, 10:47 AM:

xeger @ 45... It's allergy season

It'd appear that the nose was something of a nostrum.

#49 ::: Cadbury Moose ::: (view all by) ::: June 14, 2009, 11:01 AM:

You man need to bury Giordano Bruno more thoroughly, and ensure he is facing North (True, not Magnetic).

A rubber chicken may also be needed in case of network issues.

Cadbury (Who has only ever seen goats recommended for SCSI problems)

#50 ::: David Harmon ::: (view all by) ::: June 14, 2009, 11:04 AM:

abi #46: Well, a graceful fallback/recovery is a pretty good outcome!

#51 ::: xeger ::: (view all by) ::: June 14, 2009, 11:05 AM:

abi @ 46 ...
The congratulations must be returned with thanks. Read Patrick's note again.

The upgrade failed; we fell back to the previous version. We're going to be investigating why and trying again at another time.

Not at all! You may not have succeeded with the upgrade, but the rollback -worked-! Given the number of things I've been involved in, where the backout plan boiled down to:

"Run around, scream and shout
Hope to god we sort things out"

... I'm definitely congratulatory! You tried the upgrade... it failed... but the contingency plans (a) existed, and (b) worked!

#52 ::: xeger ::: (view all by) ::: June 14, 2009, 11:19 AM:

[paraphrased from a different event, just for an idea...]

00:00 "Hi! We're going to take the application down for maintenance. It should only take an hour, three at the worst."
02:00 "Uh, the maintenance is taking a bit longer than expected.
03:00 "We can't get the application going again"
04:00 "We think there's a problem with the database"
05:00 "There's definitely a problem with the database -- one of the tables is corrupted"
06:00 "We can't fix the table -- we need the offsite backups, and it'll take them 2 hours to get here"
08:00 "The application is still offline, and we're still waiting for the backup tapes"
09:00 "The tapes finally arrived, and we're starting to restore from backup"
10:00 "The restore is taking a long time. The next update will be in 4 hours"
14:00 "The restore is done, but we still can't get the database going. We're trying something else."
16:00 "We got the database going, but we're missing data for the last two days. We're going to try and import the missing data."
17:00 "Importing the missing data didn't go so well. We're going to stop doing that, and just bring the app up with the older data."
19:00 "The App is back up again, but missing 2.5 days of data"
19:30 "The App is down again."
20:00 "The App is up again."
21:00 "We're throttling connections to the app so it doesn't fall over again."
22:00 "We're tentatively saying the app is working for now, but we're all watching and crossing our fingers"
24:00 "The app is still running ... and we still need to import the missing data..."

[... and it went on for another day-or-so ...]

#53 ::: Teresa Nielsen Hayden ::: (view all by) ::: June 14, 2009, 11:35 AM:

Xeger (52), I'd say "Gosh, you never told me you worked for formerlyemployedatsitewhichshallremainnameless," except you said the process only went on for another day or so.

Weeks, I tell you. Months. Ongoing. And on some issues, not far short of the next plural in that sequence.

#54 ::: Earl Cooley III ::: (view all by) ::: June 14, 2009, 11:38 AM:

Cadbury Moose #49: (Who has only ever seen goats recommended for SCSI problems)

Or Ruby in a Nutshell.

#55 ::: Carrie S. ::: (view all by) ::: June 14, 2009, 11:50 AM:

Andrew* @#42: I always wanted to find a way to quickly tarnish metal, so I could tarnish the nose and polish the rest of the plaque.

*Very appropriate, that name

#56 ::: Serge ::: (view all by) ::: June 14, 2009, 11:52 AM:

xeger @ 52...

"...we need the offsite backups..."
"What offsite backups?"

#57 ::: Rob Rusick ::: (view all by) ::: June 14, 2009, 12:14 PM:

Obsxkcd: Success.

#58 ::: Rob Rusick ::: (view all by) ::: June 14, 2009, 12:15 PM:

Should have been 'Obxkcd'.

#59 ::: Jon Meltzer ::: (view all by) ::: June 14, 2009, 12:17 PM:

I was afraid of this. Standard technology advice: always double (at least) the number of goats the specs recommend.

#60 ::: P J Evans ::: (view all by) ::: June 14, 2009, 12:19 PM:

#52
That sounds like the database compression they did a couple of months ago on our stuff. It was only supposed to be down for the weekend, but it turned into most of two weeks.
Now they just run the compression every weekend, and tell us we can't do anything between Friday evening and Tuesday morning, and we hope they're right about how long it will take. (It's a big database.)

#61 ::: Jon Meltzer ::: (view all by) ::: June 14, 2009, 12:27 PM:

#56: "Okay, what about the onsite backups?"
"They're handled by another department. You need to submit a formal request."
"How long will that take?"
"No idea. The person that used to do them was an engineering coop student, and he graduated last year. I don't think he's been replaced yet."

(true story)

#62 ::: Terry Karney ::: (view all by) ::: June 14, 2009, 12:48 PM:

CarrieS: For most metals a weak acid will do the job. For copper, brass, and bronze there are specific chemicals designed to patina statues.

So apply a resist to the parts you want to keep bright, apply the acid to the whole thing. Rinse, an when the effect has the affect you like, buff to an even tone; removing the resist in the process.

#63 ::: Mary Aileen wonders if that's spam ::: (view all by) ::: June 14, 2009, 01:29 PM:

#63

#64 ::: abi ::: (view all by) ::: June 14, 2009, 01:30 PM:

It was, but I deployed the bearskin.

#65 ::: Linkmeister ::: (view all by) ::: June 14, 2009, 02:56 PM:

This does not bode well if I carry on with my plans to upgrade from MT 3.15 to MT 4. ML is a heckuva lot more complicated than my blog, though, so maybe I shouldn't read that much into it.

#66 ::: TomB ::: (view all by) ::: June 14, 2009, 03:01 PM:

My old boss liked to say "Transparent means you don't see it until you hit it."

#67 ::: joann ::: (view all by) ::: June 14, 2009, 04:25 PM:

I think something has gone ascrew with the ad column over on the right. The first link is the title of the halal cafe post, and indeed feeds back to the post via a rather circuitous route.

I don't recall seeing this sort of thing before just now?

#68 ::: janetl ::: (view all by) ::: June 14, 2009, 04:44 PM:

At WhereIWorkWhichShallRemainNameless we create and test the rollback for every upgrade. Someone will occasionally make noises about this being a waste of time for "small, low risk changes". The other people look at them with cynical expressions, and they pipe down.
I'm not a superstitious person, but somewhere down in my bone marrow, I believe that doing this is why we haven't had to do a rollback in a long time.

#69 ::: Patrick Nielsen Hayden ::: (view all by) ::: June 14, 2009, 05:04 PM:

Joann #68 -- No, actually, the "Book Blogads Network" ad includes a running feed of headlines from blogs inside blogads.com's "Books" network, and we're one of those blogs.

Blogads.com is pushing their "Books" network with a bunch of initiatives, including asking member blogs to run the ad. Since the ads they get us pay for our server expenses, we said sure.

#70 ::: joann ::: (view all by) ::: June 14, 2009, 08:33 PM:

Patrick #70:

Thanks for the explanation. I had been afraid there had been dueling stylesheets leading to messed-up tables or some such.

#71 ::: John Houghton ::: (view all by) ::: June 14, 2009, 08:41 PM:

Jon Meltzer #59 :
I was afraid of this. Standard technology advice: always double (at least) the number of goats the specs recommend.

... even when you take this rule into account.

#72 ::: xeger ::: (view all by) ::: June 14, 2009, 09:27 PM:

Jon Meltzer @ 61 ...
"No idea. The person that used to do them was an engineering coop student, and he graduated last year. I don't think he's been replaced yet."

I'll trade you the person who got their backup tapes (and mine, and another coworkers, in both cases anxiously awaited, to restore a down service), and promptly put them into their tape jukebox and overwrote them [0]...

[0] Clearly the fact that they had more tapes returned than they were expecting meant that the tapes were spare/extra, not that they -might belong to somebody else- [1]
[1] Descriptives continue to fail me, but that wasn't the most unique thing that particular coworker did...

#73 ::: Lee ::: (view all by) ::: June 14, 2009, 10:08 PM:

Carrie, #55: Liver of Sulfur.

janetl, #69: AKA "The difficulty for which one is prepared never occurs." It's not infallible, but you'd be surprised how often it appears to work.

#74 ::: dcb ::: (view all by) ::: June 15, 2009, 04:01 AM:

Re statue rubbing: in Budapest there's a bonze statue of a Hussar general. The horse he's sitting on is a stallion and its scrotum is bright and shiny, unlike the rest of the statue, becaue the local students rub this for luck before exams...

#75 ::: dcb ::: (view all by) ::: June 15, 2009, 04:03 AM:

"Bronze", not "bonze" - why do typos always become visible just after I've pressed the "Post" button?

#76 ::: Terry Karney ::: (view all by) ::: June 15, 2009, 10:32 AM:

dcb: I was working around that in ref the US Naval Acadamy.

The mascot is, "Billy the Goat". One of the allowed things to do to plebes is to say, "Billy the Goat's Balls", at which point they have to run to the goat, slap the scrotum, and run back.

#77 ::: John Houghton ::: (view all by) ::: June 15, 2009, 10:34 AM:

dcb @76:

I'm convinced that the typos don't actually exist until the POST button is pressed.

#78 ::: Xopher ::: (view all by) ::: June 15, 2009, 12:05 PM:

Maybe in the upgrade they should rename the Post button to "Reveal Typos." THAT would sure steepen up the learning curve for commenting here!

Wait, we don't want to do that, do we? I got turned around somewhere...

#79 ::: Serge ::: (view all by) ::: June 15, 2009, 12:16 PM:

Since Giordano Bruno didn't help... Has anyone thought of using Dick Giordano instead?

#80 ::: LMB MacAlister ::: (view all by) ::: June 15, 2009, 12:19 PM:

And Bruno Giordano didn't help either. But I'm happy to report he's still breathing, though somewhat dirty.

#81 ::: dcb ::: (view all by) ::: June 15, 2009, 12:43 PM:

John Houghton @ 78: I like that theory.

Terry Karney @ 77: They have to climb a bit to reach the statue in Hungary - it's on a plinth. But it's nice to see that these rituals are spread far and wide...

#82 ::: tykewriter ::: (view all by) ::: June 15, 2009, 01:40 PM:

In a cemetery in Paris is the tomb of Victor Noir, which it is considered lucky to rub. In part.

This is not the same as a head rub.

#83 ::: Rick Owens ::: (view all by) ::: June 15, 2009, 02:00 PM:

Re: backups/tests/having a plan B/other IT insurance methods and those who scoff at them...

Whenever one of our users gripes about such "wasted" efforts, I usually respond with my take on the Murphy/Finagle laws.

Long version: You're insufficiently paranoid; the universe will mess up your plans unless you do something about it, and that mess will probably be very painful to clean up. {Can be expanded as needed into an hour long lecture about Murphy, Finagle, statistics, failing gracefully v. failing spectacularly, learn from my mistakes, etc., etc. Sometimes the mere threat of such a lecture is sufficient to make folks behave.}

Short version: Which way do you want to be wrong (about your expectations)? {Because sooner or later you will be!}

#84 ::: Terry Karney ::: (view all by) ::: June 16, 2009, 07:58 PM:

dcb: Billy the Goat is on a lower plinth. I think it requires a step and jump to slap them.

Welcome to Making Light's comment section. The moderators are Avram Grumer, Teresa & Patrick Nielsen Hayden, and Abi Sutherland. Abi is the moderator most frequently onsite. She's also the kindest. Teresa is the theoretician. Are you feeling lucky?

Comments containing more than seven URLs will be held for approval. If you want to comment on a thread that's been closed, please post to the most recent "Open Thread" discussion.

You can subscribe (via RSS) to this particular comment thread. (If this option is baffling, here's a quick introduction.)

Post a comment.
(Real e-mail addresses and URLs only, please.)

HTML Tags:
<strong>Strong</strong> = Strong
<em>Emphasized</em> = Emphasized
<a href="http://www.url.com">Linked text</a> = Linked text

Spelling reference:
Tolkien. Minuscule. Gandhi. Millennium. Delany. Embarrassment. Publishers Weekly. Occurrence. Asimov. Weird. Connoisseur. Accommodate. Hierarchy. Deity. Etiquette. Pharaoh. Teresa. Its. Macdonald. Nielsen Hayden. It's. Fluorosphere. Barack. More here.















(You must preview before posting.)

Dire legal notice
Making Light copyright 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017 by Patrick & Teresa Nielsen Hayden. All rights reserved.