Back to previous post: Creative marketing

Go to Making Light's front page.

Forward to next post: Witness and response

Subscribe (via RSS) to this post's comment thread. (What does this mean? Here's a quick introduction.)

September 10, 2003

Earth Science Missions Anomaly Report: Really big owie
Posted by Teresa at 01:28 PM *

The NOAA-N Prime spacecraft being built at Lockheed Martin in Sunnyvale, CA has suffered an embarrassingly dumb accident:

Earth Science Missions Anomaly Report: GOES/POES Program/POES Project: 6 Sep 2003

DATE OF ANOMALY: September 6, 2003

LOCATION OF ANOMALY: Lockheed Martin, Sunnyvale CA

DESCRIPTION OF EVENT:

As the NOAA-N Prime spacecraft was being repositioned from vertical to horizontal on the “turn over cart” at approximately 7:15 PDT today, it slipped off the fixture, causing severe damage. (See attached photo). The 18’ long spacecraft was about 3’ off the ground when it fell.

The mishap was caused because 24 bolts were missing from a fixture in the “turn over cart”. Two errors occurred. First, technicians from another satellite program that uses the same type of “turn over cart” removed the 24 bolts from the NOAA cart on September 4 without proper documentation. Second, the NOAA team working today failed to follow the procedure to verify the configuration of the NOAA “turn over cart” since they had used it a few days earlier.

IMPACT ON PROGRAM/PROJECT AND SCHEDULE:

The shock and vibration of the fall undoubtedly caused tremendous damage. Significant rework and retest will be required. NOAA-N Prime is planned for launch in 2008.
Check out the photos. I regret to say that the aftermath looks like something out of a Warner Bros. cartoon. (via Tim Kyger)
Comments on Earth Science Missions Anomaly Report: Really big owie:
#1 ::: Emma ::: (view all by) ::: September 10, 2003, 02:23 PM:

Good golly, it looks like a Dalek accident!

#2 ::: marty ::: (view all by) ::: September 10, 2003, 03:59 PM:

Yikes! I shared it with others. Had to.

exiting, shaking my head.

#3 ::: Paula Lieberman ::: (view all by) ::: September 10, 2003, 04:44 PM:

1. Lockheed-Martin is not getting any incentive fees on that....

2. Contractor pays for that bigtime, including for any launch delay costs.... assuming that there are COMPETENT, honest, and dedicated government contract monitors involved, and not FOG whose allegiance is to the contractors being managed and their buddies, rather than the US Government and taxpayers.... Lockheed-Martin literally dropped, the way the contracts as least used to be written, that means that Lockheed-Martin gets to pay the full costs of repair, retest, and program delays. It's not a fun position for a contractor to be in, and it's a very literally unprofitable one, nothing that that sort of thing, to turn what might have been a profitable satellite program, into the red ink column.

3. If there are any competitors left, that's the sort of thing that traditionally helped move government business from the existing contractor, to someone else....

4. Embarrassing things happening in production, are more frequent in the space business than most people might realize. I remember a few from my days in the business. Sometimes the government just rolled its eyeballs if the damage could be fixed quickly and didn't cause delays or raise the costs to the government(thinking of one particular incident the contractor didn;t want the government to officially notice, and which I suppose there was something of tacit agreement on the government side that beating on the contractor would not have been at all productive... sometimes having something on the contractor to use as leverage for other stuff is useful. No, it's not blackmail, not really, especially not when there are things that the government's done that the contractor is willing to be patient on, on the other side. Adversial relationships make for really bad business.....)

Smashed up satellites from inadvertent drop tests on flight hardware, however, goes beyond anything acceptable as overlookable....

#4 ::: Ken MacLeod ::: (view all by) ::: September 10, 2003, 04:48 PM:

I find myself actually blushing in sympathetic embarrassment at the pictures. It reminds me of the time I drove a fork-lift truck into a hearse.

There is something awfully funny about the report, though.

#5 ::: Paula Lieberman ::: (view all by) ::: September 10, 2003, 04:58 PM:

"CORRECTIVE ACTION:

"Lockheed Martin formed an Accident Review Team in which GSFC is participating. The immediate actions concern safety (preventing the spacecraft from rolling, discharging the batteries, and depressurizing the propulsion system). NOAA-N Prime is under guard, all records have been impounded, and the personnel interviewed. After the safety issues are addressed, attention will focus on assessing the damage to NOAA-N Prime."

Innocuous sounding words, note that not one word above denotes "criminal investigation" "sabotage" or "adverse impact and extreme prejudice" but they're connoted.

"The shock and vibration of the fall undoubtedly caused tremendous damage. Significant rework and retest will be required. NOAA-N Prime is planned for launch in 2008."

More understatement. Were the optics mounted yet? Also, anything that depended upon structural integrity and position -- and dealing with a spacecraft that has optics and sensors fed by the optics, alignment is critical and tolerances extremely tight -- basically, that thing's going to have to be torn almost completely apart and everything checked piece part by piece part, and worse. The spacecraft that I worked with, had components that literally had requirements for at least a full year of "burn-in" testing with power on them, before they were considered flight candidate parts to be assembled onto the spacecraft.... anything with burn-in requirements on that bashed bird's going to have to undergo stability testing and characterization all over again.

The good news is that 2008 is four and a half years away -- I don't understand what takes something that's in fairly low orbit, five years of lead time, though, especially since it looked like the bird was fairly far along in assembly and test. It's awfully early to be loading it up with propellants and pressurization.... something seems quite odd to me, about that kind of lead time. I mean, there was a shorter length of time between the time the USA decided to go to Luna and when Saturn boosters started launching!

#6 ::: Lois Fundis ::: (view all by) ::: September 10, 2003, 05:25 PM:

A cross between a Dalek and C3PO!

I suspect the attention to detail signified by the invitation at the bottom of the page: "Sign up Now for our Free Newsleter".

#7 ::: lightning ::: (view all by) ::: September 10, 2003, 08:19 PM:

Ouch!

The mishap was caused because 24 bolts were missing from a fixture in the 93turn over cart94. Two errors occurred. First, technicians from another satellite program that uses the same type of 93turn over cart94 removed the 24 bolts from the NOAA cart on September 4 without proper documentation. Second, the NOAA team working today failed to follow the procedure to verify the configuration of the NOAA 93turn over cart94 since they had used it a few days earlier.

In other words, somebody "borrowed" a couple dozen critical bolts and the crew didn't check them.

Somebody's gonna be in deep, deep trouble over this. I just hope it's the right person/people, rather than a convienent scapegoat. (When I worked for the military, embarrasing mistakes were invariably blamed on either a secretary or a draftsman.)

#8 ::: Bruce Arthurs ::: (view all by) ::: September 11, 2003, 12:17 AM:

So, if the other team removed the 24 bolts... why? Was -their- turnover cart missing numerous bolts? And if so... why?

It's pretty clear that the satellite got smashed up because its team failed to doublecheck the turnover cart. Regrettable, but...

...but it's that other team, who cannibalized parts from the satellite team's equipment, that seems to be a symptom of MUCH bigger problems. Is there a bolt shortage? Why? Are other parts in the same situation? Why?

#9 ::: Jordin Kare ::: (view all by) ::: September 11, 2003, 01:52 AM:

"Really big owie" is right. Think, oh, 50 or 60 top-of-the-line Ferraris simultaneously being dropped from a high place. My stomach hurts just looking at it -- makes me think of the time in college when I managed to pull a rather large laser off an optical table onto the floor, multiplied by a few thousand.

I keep imagining the thoughts going thru the head of the poor guy with his finger on the cart lift button as he watched the satellite sli-i-ide and ti-i-ip. They say the NTSB finds the last word on most recovered cockpit voice recorders is "Shit!"

BTW, POES is a series of polar orbiting weather satellites. You can see the POES program plan here
There are supposed to be 2 flying at a time, and each one lasts 4-5 years. NASA bought the last 4 (POES-L, M, N, and N') all at once; they'll be replaced by a new series (NPOESS) after N'.

N' is basically a spare; L and M have launched, and N is scheduled for next June, 2 years after M, but N' was obviously ordered in case one of the others failed. N' was scheduled to be available to launch any time after late '04. As long as N works, and M lives a normal lifespan, they have plenty of time to fix this one; if N goes in the drink, Lockheed will have to scramble.

The bad news is, it looks like they were ahead of schedule in integrating N'; except for solar panels, it looks like the bird is largely complete. The good news, if any, is that it landed with most of the instruments on top. Using an umpteen million dollar spacecraft bus as a crumple zone is not something one tries to do, but it's better than landing with the instruments on the bottom...

#10 ::: Steve Taylor ::: (view all by) ::: September 11, 2003, 02:55 AM:

Awe inspiring... I think I need to print one of the pictures out to hang over my desk, to remind me how happy I am that my work mistakes can't cause this much damage. I could probably lose my employer a few hundred thousand if I did something exceptionally stupid - but nothing on this scale.

Very nice to see the comments from people who've worked in the industry.

#11 ::: David Goldfarb ::: (view all by) ::: September 11, 2003, 03:45 AM:

The first place that I saw this one was as one of the Particles on Electrolite.

#12 ::: bad Jim ::: (view all by) ::: September 11, 2003, 05:07 AM:

Team B probably didn't borrow the 24 missing bolts. Perhaps they started to replace the fixture with a different one but stopped, for some reason. Team A, seeing the original fixture in place, neglected to verify that all the bolts were still inserted.

Sabotage by Team B can't be ruled out.

#13 ::: Jon H ::: (view all by) ::: September 11, 2003, 09:17 AM:

Jordin Kare writes: ""Really big owie" is right. Think, oh, 50 or 60 top-of-the-line Ferraris simultaneously being dropped from a high place."

Or like that shipload of BMWs, Volvos, and Saabs that went down in the English Channel.

If you look closely, you can see some of the cars, pretty banged up, in this picture of a recovered section of the ship.

#14 ::: Harriet Culver ::: (view all by) ::: September 11, 2003, 12:23 PM:

Clearly it's not a parallel situation, but was I the only person whose thoughts turned irresistably to Mme Ekaterin Vorsoisson upon first seeing this linked from Particles the other day? :-)

#15 ::: Robert Brazile ::: (view all by) ::: September 11, 2003, 01:31 PM:

Unfortunate though it is, my mind was irresistably drawn to the Watterson line (and book title)

Scientific Progress Goes "Boink"

#16 ::: Jordin Kare ::: (view all by) ::: September 11, 2003, 05:19 PM:

>Scientific Progress Goes "Boink"

I prefer the classic line, "Experience is directly proportional to equipment ruined." I'd say Lockheed-Martin can now proudly claim to have some of the most experienced technicians in the industry.

#17 ::: Steve Taylor ::: (view all by) ::: September 11, 2003, 07:37 PM:

> was I the only person whose thoughts turned
> irresistably to Mme Ekaterin Vorsoisson upon
> first seeing this

Never occured to me until you said it, but my God, yes!

#18 ::: Brooks Moses ::: (view all by) ::: September 13, 2003, 04:38 AM:

From the description of things, it would almost sound as if these were some bolts that might be reasonable to remove, and that one might almost reasonably not notice were missing in the process of moving things. At least, that's how I read it.

Then I looked at the picture. This satellite is mounted on a six-foot-diameter metal pad, and this pad is mounted on a corresponding pad on the cart, and there are about 36 currently-empty bolt holes that connect the two. (My guess is that some of them weren't used.)

This means that somebody was standing next to this large and undoubtably heavy spacecraft that is much taller than they are, and was removing every single bolt that was holding it up, leaving it some approximation of merely balanced in place. And these are certainly Not Small Bolts.

I cannot honestly imagine that removing these bolts without otherwise securing the satellite was either safe or included in the allowable proceedures, regardless of documenting that it had been done afterwards.

#19 ::: andrew ::: (view all by) ::: September 13, 2003, 05:19 AM:

Funny!

If I worked there, I'd have to suppress the giggles. That's probably why I've never been entrusted with anything valuable.
Once, though, I forgot to set the emergency brake on a big delivery truck that I was driving...and while I was placidly eating my lunch, the truck rolled out into the street - smashing itself, an innocent car, and part of the truck's payload of automotive windshields.
I drove the truck back to the office, parked it in the lot, went immediately home and didn't answer my phone for two weeks.

Oh, and another time, when I was working at a plant nursery, I accidentally turned on the overhead sprinkler system when the place had a full load of customers, soaking the lot of them. Yes I did.

So spaceship droppers, take heart. We all make mistakes.

#20 ::: tavella ::: (view all by) ::: September 13, 2003, 04:32 PM:

I have to agree with Brooks; if the bolts were indeed 'borrowed' by another team, it looks like something a lot closer to sabotage than an accident.

I'm wondering if instead, it was never bolted in in the first place? Mind you, anyone who lowered the spacecraft into place and didn't bold it in would be negligent, but it seems more likely than someone going to all the effort of unbolting them.

#21 ::: Xopher ::: (view all by) ::: September 14, 2003, 12:53 AM:

Harriet - [smacks forehead] That's IT! That's why this sounded familiar.

Thanks, that was driving me crazy.

#22 ::: Morat ::: (view all by) ::: September 16, 2003, 04:08 PM:

I work for LM. This "accident" came up in several meetings. The general consensus? "Thank God I don't work in that division.".

Two major mistakes: Removing the bolts (big no-no) and failing to check them before moving (big no-no). Both cases, someone willfully disregarded procedure. They'll be lucky to be merely fired.

#23 ::: Morat ::: (view all by) ::: September 16, 2003, 04:10 PM:

From what came up in staff, by the way, it was bolted in place before. After attaching (and bolting) the sattellite, they took a break (lunch, overnight, it wasn't clear) and were going to move it "later". Merely attaching the satellite to the truck is a several-hour job.

#24 ::: Jim Gillingham ::: (view all by) ::: September 16, 2003, 09:03 PM:

I have an image of somebody going to their boss - "things would go a lot quicker around here if we had a set of bolts for each cart" and the manager decides to save $100 or so.

#25 ::: Jon H ::: (view all by) ::: September 16, 2003, 09:46 PM:

Somewhere, there's a Robot Wars robot with some really bitchin' protective metal bolts all over it.

#26 ::: Joe ::: (view all by) ::: September 25, 2003, 03:35 PM:

The poor baby was essentially complete, with all of the major instruments loaded. You can see the HIRS, AMSU-A1, MHS and AMSU-A2 in the side view picture. The AVHRR is hidden behind the HIRS. NOAA-N can be seen in the background. Since the AVHRR and HIRS are Cassegrain telescopes, you can guess what happened to the optics.

Note that the factory is in Sunnyvale. Sunnyvale is in California, which has earthquakes. Leaving a $239M spacecraft unsecured in an earthquake zone is beyond irresponsible. They should never have sold the East Winsor New Jersey factory.

The immediate problem is how to move N' so that N can be transported to Vandenberg for launch next September.

BTW... this buy was for KLM, with N and N' added on. It is called N' rather than O because OPQ was intended to be the next block buy until NPOESS came along.

Choose:
Smaller type (our default)
Larger type
Even larger type, with serifs

Dire legal notice
Making Light copyright 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017 by Patrick & Teresa Nielsen Hayden. All rights reserved.